Skip to main content
Download PDF
Download ePub

Analyzing the usage of theories of change for routine immunization programs -- a review of impact evaluations from LMICs

Journal of Health, Population and Nutrition volume 43, Article number: 141 (2024) Cite this article

Abstract

Background

In this article we analyzed the extent of the usage of Theories of Change (TOCs) and causal pathways in the evaluation of immunization programs to identify the challenges to generating evidence on how interventions improve immunization.

Methods

We analyzed the use of the TOC in impact evaluations (IEs) of immunization interventions published after 2010, and its associated articles. The review includes studies from Evidence Gap Map and Yale review that were conducted in May and March of 2020, respectively. We synthesized data on six domains using NVIVO — program theory, context, assumptions, usage of TOC, use in evaluation, and description causal pathways.

Results

Our review included 47 large-scale and 45 small-to medium-scale interventions. Of the included studies, 19% used a TOC, 56% described a causal pathway or used a conceptual diagram with varying degrees of detail, and 25% of the IEs did not provide any information on how their intervention was expected to affect change. Only 19 of the 92 IEs explicitly outlined any assumptions associated with the implementation of the interventions. Forty studies measured the outputs or intermediate outcomes leading to improved immunization coverage.

Conclusion

Future implementers and evaluators need to develop clear TOCs that are based on established theory and have clearly articulated the underlying assumptions. Large-scale health system strengthening initiatives implemented by governments, also need to build TOCs and integrate them into their results frameworks. Additionally, there is a need to combine both impact and process evaluations to understand the how context affects the causal pathways.

Introduction

With the increasing need to scale up immunization programs in diverse contexts, there is a rise in the demand for evidence on how interventions work and why [1,2,3]. The literature contains reviews that assess whether strategies such as reminders/recalls [4], education of mothers [5, 6], m-health interventions [7,8,9], or community mobilization interventions [10] are effective in improving routine childhood immunization outcomes. While there is extensive evidence on whether programs created an impact, there are gaps in our knowledge of how that impact was achieved. Consequently, there is a renewed focus at the international level on exploring pathways of change using an iterative, learning approach to monitoring and evaluation, with context-specific actionable data [11, 12]. Guidelines for evaluating complex interventions underline the need for understanding the causal mechanisms underpinning the intervention and for basing the evaluation on it to build an evidence base that informs policy [13].

Several tools such as the Theory of Change (TOC) have been used to depict the causal mechanisms through which programs are expected to create impact. Developing TOC is an exercise of drawing a diagram that links impact and the relevant inputs and intermediate outcomes and has gained prominence with the rise of complex, multilevel interventions in public health [14, 15]. With increasing complexity, TOCs were introduced to address the limitations of other tools such as the logic models and results frameworks. The logic models served more as a descriptive tool mandated by funding agencies and donors than as an explanatory tool to understand why the activities led to change. In a review, logic models were said to have a ‘missing middle’ that does not explain how outputs and intermediate outcomes translate to long-term outcomes [14]. Additionally, they did not facilitate critical thinking and learning as context, and the inherent beliefs and assumptions underpinning the program were not majorly accounted for [3, 16]. Since its initial use in the early 1990s, TOCs have been used as an implementation tool to aid program development, internal organizational communication, external dissemination and advocacy, program adaptation and learning, and more importantly, to determine how we evaluate program implementation and its effectiveness [14, 17,18,19].

In this manuscript, we present results from the analysis of the use of TOCs by reviewing published impact evaluations and relevant supplementary papers on immunization interventions in low- and middle-income countries (LMICs). We also explored how TOCs and causal pathways were translated into evaluation approaches for these immunization programs to understand the gaps in evidence on how interventions improve immunization uptake. We aimed to answer the following questions on the use of TOC for immunization interventions: a)Where and how were TOCs used b) How have TOCs and causal pathways translated into monitoring and evaluation strategies, and what are the implications for evidence on how programs work?

Methods

Criteria for selecting impact evaluations in this review

Evidence bases used for the search

We conducted this review of impact evaluations (IEs) from a representative sample of countries from LMICs. We searched for studies from the Evidence Gap Map (EGM) on routine childhood immunization [20] and the review conducted by the Yale Institute for Global Health [21] (Refer to Fig. 1 for details). The last search of EGM and Yale review were conducted in May and March of 2020, respectively.

Fig. 1
figure 1

Methods – search and inclusion of studies

Full size image

Inclusion criteria

We identified impact evaluations from select WHO (2022) and GAVI (2022) priority countries from both the EGM and Yale reviews based on three main criteria: (i) countries that have stagnant or low vaccination rates and (ii) those that have a sizeable evidence base (at least 3–4 studies) for a given country for synthesis of evidence. If three studies were not found from one of the three regions (Sub-Saharan Africa, South and South East Asia, or Latin America), countries with fewer studies were included. iii) Of the countries in each of these three regions, we selected one country that has a high routine child immunization coverage rate. Impact evaluations were included from the following 11 countries: India, Indonesia, Bangladesh, and Pakistan from the South and South East Asia regions; Ethiopia, Rwanda, and Nigeria from the African region; and Brazil, Nicaragua, Columbia, and Guatemala from the Latin American region.

We selected all interventions that impacted the routine immunization coverage of children aged 0–5 years or any intermediate outcome as causal factors. We only included experimental and quasi-experimental studies with counterfactuals published on or after 2010 to determine the causal impact of an intervention in comparison to standard or usual care.

Identification of secondary /qualitative studies

We identified feasibility studies, qualitative studies, formative or process evaluations, cross-sectional or observational studies, project documents or funder reports, and policy briefs related to the IEs from clinical trial registries and Google Scholar. In addition, we also carried out targeted searches of the websites of implementing agencies and funders. We also identified studies about the study area of the IE, discussing the intervention in question, if it was conducted in or referring to the year in which the intervention was implemented.

Review and analysis of the use of theory of change

We reviewed the above IEs to identify explicit usage of TOCs, program theories, logic frameworks, conceptual diagrams, and explanations of causal pathways of the intervention written in the form of intended impact or rationale for the program. We analyzed and extracted data from the identified IEs and their supporting documents using a list of questions (Refer to Table 1 for details). This list was populated based on the report on TOCs commissioned by Comic Relief and UKAID in 2011 [14, 17]. The questions also draw from the core principles identified by Coryn and his colleagues in their review on the use of program theory for evaluation and Medical Research Council guidance on evaluating complex interventions [22,23,24]. The studies were coded using NVIVO software to identify any text on TOCs, causal pathways, or the mechanisms of action of the programs. Attributes such as the year of implementation, scale of implementation, and use of program theory in designing programs were also captured. We defined large-scale interventions/programs implemented at the state or national level including studies implemented in many districts. All the other interventions were categorized as small- to medium-scale interventions.

Table 1 List of questions guiding the data extraction for the review
Full size table

The TOCs were classified in terms of scope and detail as Broad/Narrow or Deep/Shallow based on the work of ActKnowledge on TOCs [16]. In the context of immunization programs, pathways of broad/deep nature identify all the outputs and intermediate outcomes that can cause improvements in immunization coverage and identify all the pathways that can explain what, how, and why change has occurred. Narrow and shallow TOCs focus on none or a few intermediate outcomes affecting the outcome of the intervention. TOCs are classified as shallow TOCs if they do not provide any detail.

The data extracted for this review were analyzed to identify the extent of the usage of TOCs and how TOCs or causal pathways were translated into monitoring & evaluation. We present the results of the analysis using tables and matrices.

Results

The study reviewed 92 IEs along with 93 related articles to analyze the usage of TOCs (Refer to the Supplementary File 1). The studies evaluated different types of intervention strategies that were used either singly or in combination with other strategies. Of the 92 IEs, 47 were large-scale interventions implemented at the national or state level, and 45 were small- to medium-scale interventions (Refer to Table 2 for details). Forty-six interventions were implemented before the year 2010, while the rest were implemented on or after 2010. Most of the included studies did not have a TOC or described causal pathways in text with varying degrees of detail; hence, most of the studies were classified as narrow or shallow. A total of 19.56% of the studies used a theory of change, 56.52% described a causal pathway or used a conceptual diagram with varying degrees of detail, and 25% of the studies did not provide any information on how their intervention was expected to affect change. Only 19 of the 92 IEs explicitly outlined any assumptions associated with the implementation of the interventions. Of the interventions included in these 92 studies, 37 were single-component interventions, while 55 were multicomponent interventions. Forty interventions measured one or more of the outputs or intermediate outcomes leading to improved immunization coverage.

Table 2 TOC usage vis-a-vis characteristics of impact evaluations
Full size table

For this manuscript, we divided the results section into findings about large-scale interventions and small- to medium-scale programs and analyzed them considering the type of TOC, year of implementation, type of intervention, and TOC content (Refer to supplementary material 2 for details of data extracted from the included impact evaluations).

Large-scale programs

Most of the large-scale interventions were started before 2010 and involved an equal number of single and multicomponent programs (Refer to Table 3 for details). The large-scale interventions were implemented by governments or in partnership with governments using multiple strategies.

Table 3 Description of the intervention included in the review
Full size table

Theory of Change: Most of the evaluations of the large-scale interventions did not provide any TOC diagram. Only three of the large-scale interventions had a detailed TOC explaining the various inputs, outputs, and intermediate outcomes [32, 34, 67]. Thirteen of the forty-seven large-scale programs had no details on how the intervention was expected to have an impact (Refer to Table 2 for details).

The authors of 33 of the 47 IEs of large-scale programs provided details of the causal pathways in the text in varying degrees of detail, by referring to the literature in the area or through program theories (refer to Table 4 for details). The descriptions of the pathways of change can be described as shallow, as they provided little information on how these macro-level changes create impact. The authors of evaluations of these nationally implemented programs relied on the literature to explain how these programs could improve immunization outcomes.

Table 4 Examples of interventions providing causal pathways
Full size table

For example, Aggarwal cited studies that showed how access to all-weather roads could improve access to healthcare services by reducing transport costs, improving employment opportunities, and thereby improving incomes and increasing awareness through improved social interaction [113]. Kusuma et al. cited theories exploring how Conditional Cash Tranfers (CCT) could incentivize behaviors. The authors cited the human capital theory, which posits that consumers will invest in health if the expected private benefit exceeds the cost. In terms of vaccination, the cash element of the CCT was thought to help with financial barriers, and the conditionality element was shown to transfer health information on the benefit of vaccination and signal the importance of vaccination for both households and health workers [52]. The authors also used conceptual diagrams depicting the program elements and their links with outcomes [57].

Assumptions: Notably, in only 12 of the studies, the underlying assumptions were outlined in varying degrees of detail. For example, Okeke et al. noted that implementers assumed that the program would be effectively implemented without any roadblocks. Some authors have discussed the conditions that have been assumed to exist, in the causal pathways. Cristia et al. emphasize that there is a basic assumption about the number of potential providers in a market that would allow for competitive bidding [93]. Talukdar et al. assumed that the providers of health services would be aware of the voucher program [99]. In their evaluation of the performance-based financing (PBF), Sato and Belel assumed that stockouts and absenteeism were barriers to vaccination uptake [97]. However, largely, the interventions provided little to no information on the assumptions underlying the program.

Context: Many of these large-scale programs were implemented by the government and were rolled out at the national or state level. Understandably, the context considered for these programs has been the broader national healthcare context, such as the burden of disease, the lack of coverage of services, or the critical bottlenecks in the health system in general. The local context was not considered in many of these evaluations, which used national surveys to assess these programs.

Use of program theory or theoretical frameworks: Seven of the large-scale interventions were based on an established program theory. Examples include the PROCEED-PRECEED framework for designing interventions, human capital theory, and the social justice theory for conditional cash transfers. Many of the large-scale interventions did not cite any program theory.

Integration of TOC/Causal pathways in the evaluation of Large-Scale Interventions: In 14 of the evaluations of the large-scale programs, one or more of the outputs and intermediate outcomes on the causal pathway were assessed that led to improvements in vaccination outcomes. Our review showed that of the three evaluations with a TOC, two measured intermediate outcomes and outputs, leading to improved vaccination uptake. For example, in the evaluation of the Midwife Service Scheme in Nigeria, Okeke et al. explored how the intervention affects mothers’ knowledge levels, access to quality care, and perceptions of quality [32]. Saggurti et al., in evaluating health intervention integration with women’s groups, measured the link between the intervention and the extent of collectivization [34].

Of the 33 studies that described only the causal pathway, 12 measured intermediate outcomes. For example, in their evaluation of the voucher scheme in Bangladesh, Talukdar et al. measured improvements in the quality of health services, reductions in out -of-pocket expenditures, and increases in beneficiary awareness of health issues [99]. In an evaluation of the impact of the rural electrification program, Jyotigram Yojana in Gujarat, Chen et al. assessed the impact of the availability of continuous electricity on cold chain maintenance which could in turn affect vaccination outcomes [115]. However, most of the evaluations did not assess intervention’s impact on intermediate outcomes (refer to Fig. 2 for details).

Fig. 2
figure 2

Use of TOC in monitoring and evaluation

Full size image

Small- to medium-scale intervention in immunization

There were 45 small- to medium-scale interventions that were included in this review. Unlike large-scale interventions, which started before 2010, most small- to medium-scale interventions started after 2010 and included twice the number of multicomponent interventions as single-component interventions (Refer to Table 2 for details).

Theory of Change Diagram: Compared to large-scale interventions, the use of TOC is substantially greater in this category. Of the 45 interventions, 15 studies used a TOC diagram to explain how the intervention was likely to cause impact, while 18 of them described the causal pathway in the text. There are 13 TOCs in multicomponent programs compared to two single-component programs. Among the programs that used a detailed TOC were complex multicomponent community mobilization interventions [117, 118].

Eighteen of the evaluations of the interventions described a causal pathway in the text (Refer to Table 4 for details). Many of the interventions involved reminders/recalls or education interventions, and the cited literature that links education and reminders with improved knowledge and awareness among mothers [37, 45]. For community mobilization interventions, the authors hypothesized the pathway of impact. Sengupta et al. hypothesized that if outreach immunization clinics and community guardians were made available, then access to immunization would increase, thereby increasing childhood vaccination uptake [104]. Twelve interventions did not report any causal pathway in any form.

Assumptions: Only seven studies explicitly stated the assumptions underlying the interventions. Of these, six of the impact evaluations had a detailed TOC. For example, Adamu et al. stated that the intervention assumed that missed opportunities for vaccination rates would remain unchanged in the absence of the intervention [88]. Banerjee et al., in the evaluation of their intervention on incentives for immunization, reported being based on the assumptions that ASHAs (community health workers) were aware that the caregivers received the incentives, mobile recharge incentives were valued, and caregivers trusted and valued ASHAs [61]. Most of the studies that listed the underlying assumptions of the program had a detailed TOC.

Context: In many of the small- to medium-scale programs, the social, geographical, demographic, cultural, and health system context in which the program was being implemented was described. The context was also cited to provide a rationale for implementing the program in the region. For example, IEs provided a detailed profile of immunization rates in a region along with the socioeconomic profile to show the lack of resources and poor health in a certain context. In some cases, such as the team-based goals and incentives program in Bihar, India, a feasibility study was performed to assess the acceptability of the program in a certain region [107, 119].

The context was also discussed to explain the evaluation results or any unanticipated changes that affected the program or evaluation’s implementation. For example, Domek et al., in the evaluation of the text message reminder intervention in Guatemala, wrote that ‘political instability that led to unusually high levels of vaccine shortages affected the study and the team’s ability to track specific vaccine completion rates’ [70]. Wallace et al. found that ‘healthcare providers in the Indonesian Ministry of Health did not have standard protocols in place on how to remind parents about future vaccination visits, and that providers followed a variety of localized practices, including creating their own written reminders’ [80]. The context was discussed in interventions that had a feasibility study or a process evaluation as a part of the impact evaluation of the intervention.

Program theory and theoretical framework: Nine interventions cited a program theory or model underlying the interventions. The theories include diffusion theory, social cognitive theory, motivation theory, the framing of health theory, the health belief model, and the COM-B model.

Integration of TOC/Causal pathways in the evaluation of small- to medium-scale interventions: Evaluations of twenty-six of the forty-five small- to medium-scale interventions measured the outputs or intermediate outcomes leading to improved immunization outcomes (Refer to Fig. 2 for details). Of the programs with a TOC, 11 impact evaluations measured the outputs and intermediate outcomes that could affect immunization uptake. For example, Oyo-Ita et al., in their evaluation of the program on engaging with traditional religious leaders, measured the degree of community engagement; qualitative assessment of knowledge, attitudes, and beliefs; and satisfaction with TRLs [117].

Of those without a TOC but with a causal pathway, 13 had one or more intermediate outcomes. For example, interventions that focused on training or educating mothers about child health practices using face-to-face interactions or m-health interventions measured the knowledge levels among mothers [30, 36, 37, 43, 45, 75, 78]. Manyazewal et al. measured process indicators such as human resources, documentation, and supply for quality improvement initiatives implemented in Ethiopia [90].

Discussion

Our review showed that TOCs have not been used in the evaluation of large-scale programs, particularly health system strengthening interventions. TOC usage is more common in small- to medium-scale programs and multicomponent interventions and has increased since 2010. We also find that there is a weak integration of TOCs/causal pathways into many evaluations, which adds to the evidence gap on what we know about how and why interventions work. Additionally, aspects such as the use of program theory and the articulation of assumptions underpinning program implementation have been sparse, with only programs with a detailed TOC including these aspects in the evaluation. Our review also revealed that context has been mostly used to provide a background for the program—except in instances where a detailed TOC is provided—or as a part of the process evaluation performed alongside the impact evaluation.

Strengths and limitations

Our review is one of the few research publications on the use of TOCs in the context of routine immunization interventions. While there can be disagreements about the extent of the evidence included in this review, our analysis provides an initial understanding of the use of TOCs in the evaluation of immunization programs. With the increase in the use of theory-based evaluation techniques and the acknowledgment of newer approaches to evaluating complex interventions, much of the literature on TOCs is dominated by evidence on the process of developing TOCs [1, 120,121,122,124], the synthesis of multiple theories of change [125], or the validation of TOCs by checking whether they are consistent with the literature [126, 127]. Our review adds to the small body of evidence examining the extent of the use of TOC in a specific area. Other such reviews include a scoping review on the use of TOC for child health interventions [128], a systematic review examining the use of TOCs in public health in general [129], and a review to understand the use of TOCs in the development sector [14]. Unlike these reviews that focused mainly on high-income contexts, ours is the first to cover a large evidence base from 11 low- and middle-income countries in South and South East Asia, Latin America, and Africa in routine immunization.

It should be noted that our review is based on impact evaluations and relevant program documents identified for another review on immunization and its drivers. The analysis is limited by what is reported in impact evaluations, specifically for large-scale programs that do not have a detailed TOC. Additionally, the review included evaluations conducted at the national level and did not include subnational process evaluations or qualitative studies related to nationally implemented large-scale programs. We also acknowledge that we conducted this analysis on studies from a representative sample of countries included from the EGM and Yale review which were conducted in the year 2020.

Our review has the following policy and research implications for immunization programs:

  1. 1.

    Evidence on how large-scale health system strengthening programs affect immunization has limited use for scalability without a detailed TOC

Many of the evaluations of large-scale interventions, specifically the Health System Strengthening (HSS) interventions included in this review, did not include a broad TOC to explain how the intervention strategies could affect immunization. The review showed that in many large-scale HSS programs, evaluators had taken a linear and siloed view of how the various health system strategies work. Recently, there has been an increasing focus on taking a systems approach to evaluate HSS interventions [130]. The literature in the HSS has shown a gradual move from a view where theories of change exist for individual health system building blocks to a view where health system changes are viewed as interventions influencing the whole system [130, 131].

Our results agree with the review by the ReBuild and ReSYST Consortia for FCDO in 2021, where authors reviewed 96 HSS studies to understand the state of evidence [132]. The FCDO review showed that for HSS interventions focusing on service delivery integration, health financing, and logistics, evidence needs to go ‘beyond the effectiveness of an HSS intervention on service delivery, toward gaining an understanding of the processes and expected mechanisms of change’. For health financing interventions that facilitate functions within various health system elements, the review revealed that studies need to explore the underlying pathways of change, as several different mechanisms of action can result in the same health system effects.

In the context of immunization, using TOCs to understand how health system changes influence the wider system has greater importance, as routine immunization services are integrated with other child and development services, and can be influenced by other supplementary vaccination programs for emerging infections such as COVID-19. They can also depend equally on the demand side of the intervention.

  1. 2.

    Integration of TOC into evaluation needs to use multiple methods and approaches

Our review showed that the lack of a detailed TOC meant that there was a lack of or weak integration of TOC in the approaches used to evaluate the programs. Many of the evaluations did not measure any outputs or intermediate outcomes leading to improvements in immunization uptake. Evaluators took an impact evaluation approach for many of these large-scale interventions, especially HSS. They used quasi-experimental designs with secondary data from nationally administered surveys or routine administrative data to evaluate these programs. Few of them supplemented the findings of the results with concurrent process evaluations or used mixed methods. Consequently, the HSS intervention outputs—human resources, supervision, training, or infrastructure—could not be assessed to determine how the intervention influenced these aspects. The spillover effects of HSS interventions on other health system components were also not measured. Finally, given that nationally administered survey data were used to assess impact, the subnational context could not be accounted for when interpreting the results of the evaluation. Multicomponent interventions that operate at the familial, community, and health system levels did not measure many causal links between the intervention and immunization outcomes.

There is a need to use mixed methods and novel approaches to better understand how interventions cause change. In the context of HSS interventions, the FCDO review showed that for programs improving human resources and skills, the challenge is to understand which combination of strategies works and how, pushing research towards exploring other fields, such as organizational development, to answer these questions [132]. Process evaluations have gained prominence for evaluating complex programs with a specific focus on implementation aspects such as fidelity, adaptation, and dose, as well as mechanisms of impact [133]. Process and impact evaluation approaches are being integrated to better understand if an intervention works and how [23]. Future research needs to leverage multiple approaches and use both qualitative and quantitative methods for evaluation.

  1. 3.

    Evaluations need to be based on program theories

Program theories and frameworks are used to create a shared understanding of how a program is expected to have an impact in a certain context [134]. Our review showed that in most of the interventions, any theory or framework underpinning the intervention was not cited. The lack of a TOC and any relevant theory informing the intervention strategy adds to the gap in our understanding of how programs are expected to make an impact on routine immunization. It also prevents the synthesis of evidence on what worked and how in the context of existing theories.

  1. 4.

    Context needs to be accounted in the design and evaluation of programs

The review has also shown that context is mostly used to provide only a background or rationale for the intervention and to explain the challenges in the implementation or evaluation of the programs in a few studies. Overall, the descriptions of the context were cursory and did not adequately specify which features of the context were relevant or were most significant to the intervention or its delivery.

In the case of HSS interventions, there are stark differences in health system responsiveness and readiness across different contexts. Understanding why and how an intervention performs in a certain context as opposed to another requires constant adaptation and learning from evaluations. TOC helps identify and elaborate on the critical components that could be tested using implementation research techniques to adapt or drop across contexts. Furthermore, the availability of a TOC can help address the persistent challenge of balancing program fidelity and adaptation in newer contexts. This approach might enable some flexibility in intervention implementation and thereby its transferability to different contexts [134, 135]. In 2018, UNICEF, in partnership with GAVI, funded its implementation research for immunization programs in LMICs to understand what contextual and programmatic factors influence program implementation and why [136]. Context has also gained importance in learning systems to adapt programs according to the context and lessons learned [137].

There is a need to move away from looking at interventions as separate packages of components that are introduced in a certain context. In developing and justifying a theory of change to inform an intervention and its evaluation, researchers should show a clear understanding of how the context influences the program and vice-versa [13]. An intervention that is effective in some settings could be ineffective or even harmful elsewhere [13, 24, 138]. Thus, it will be beneficial to prioritize the analysis of heterogeneous and unintended effects in the evaluation depending on the usefulness of information for decision-making.

Conclusion

Our review highlights the strong need to strengthen theory-based evaluation approaches for the routine immunization of children in LMICs. Future implementers and evaluators need to develop clear TOCs that are based on established theory and have clearly articulated the underlying assumptions. Importantly, for the evidence to be usable for policy or practice, the lack of evidence on causal mechanisms can be a major roadblock. An increase in complex programs also require novel approaches and the use of multiple evaluation methods. Large-scale programs, specifically health system strengthening initiatives implemented by the governments, also need to focus on building TOCs and integrating them into their results frameworks. There is a need to combine both impact and process evaluations and to use implementation research techniques to understand how the intervention has affected the outputs and outcomes that can impact immunization uptake.

Data availability

Data is provided within the manuscript or supplementary information files.

Abbreviations

CCT:

Conditional Cash Transfer

EGM:

Evidence Gap Map

GAVI:

Global Alliance for Vaccines and Immunization

HSS:

Health System Strengthening

IEs:

Impact evaluations

LMICs:

Low- and middle-income countries

m-Health:

Mobile Health Interventions

M&E:

Monitoring and Evaluation

MOV:

Missed Opportunities for Vaccination

TOC:

Theory of Change

WHO:

World Health Organization

References

  1. Hailemariam M, Fekadu A, Selamu M, Alem A, Medhin G, Giorgis TW, et al. Developing a mental health care plan in a low resource setting: the Theory of Change approach. BMC Health Serv Res. 2015;15(1).

  2. Rasheed MA. Scaling nurturing care interventions in the health sector: a Theory of Change perspective. Front Public Health. 2022;10:903342.

  3. De Silva MJ, Breuer E, Lee L, Asher L, Chowdhary N, Lund C, et al. Theory of Change: a theory-driven approach to enhance the Medical Research Council’s framework for complex interventions. Trials. 2014;15(1).

  4. Jacobson Vann JC, Jacobson RM, Coyne-Beasley T, Asafu-Adjei JK, Szilagyi PG. Patient reminder and recall interventions to improve immunization rates. In: Cochrane database of systematic reviews. Vol. 2018. John Wiley and Sons Ltd; 2018.

  5. Saeterdal I, Lewin S, Austvoll-Dahlgren A, Glenton C, Munabi-Babigumira S. Interventions aimed at communities to inform and/or educate about early childhood vaccination. In: Cochrane database of systematic reviews. Vol. 2014. John Wiley and Sons Ltd; 2014.

  6. Kaufman J, Ryan R, Walsh L, Horey D, Leask J, Robinson P, et al. Face-to-face interventions for informing or educating parents about early childhood vaccination. In: Cochrane database of systematic reviews. Vol. 2018. John Wiley and Sons Ltd; 2018.

  7. Dumit EM, Novillo-Ortiz D, Contreras M, Velandia M, Danovaro-Holliday MC. The use of eHealth with immunizations: an overview of systematic reviews. Vaccine. 2018;36(52):7923–8.

  8. Abdulrahman SA, Olaosebikan MO. mHealth: a narrative synthesis of evidence of its application in improving childhood immunization coverage. J Hospital Manage Health Policy. 2017;1(4). https://doi.org/10.21037/jhmhp.2017.10.01

  9. Watterson JL, Walsh J, Madeka I. Using mHealth to improve usage of antenatal care, postnatal care, and immunization: a systematic review of the literature. Biomed Res Int. 2015;2015.

  10. Jain M, Shisler S, Lane C, Bagai A, Brown E, Engelbert M. Use of community engagement interventions to improve child immunisation in low-income and middle-income countries: a systematic review and meta-analysis. BMJ Open. 2022;12(11):e061568.

  11. World Health Organization. IMMUNIZATION AGENDA 2030. 2020.

  12. Decouttere C, De Boeck K, Vandaele N. Advancing sustainable development goals through immunization: a literature review. Global Health. 2021;17(1):95. https://doi.org/10.1186/s12992-021-00745-w

  13. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015;350:h1258. https://doi.org/10.1136/bmj.h1258

  14. Vogel I. Review of the use of “Theory of Change” in international development Review Report. 2012. https://www.isabelvogel.co.uk/

  15. Taplin DH, Clark H. Theory of Change basics - a primer on Theory of Change. 2012. https://www.actknowledge.org/

  16. Taplin DH, Clark H, Collins E, Colby DC. A series of papers to support development of theories of change based on practice in the field. 2013. https://www.actknowledge.org/

  17. James C. Theory of Change - review. 2011.

  18. Stein D, Valters C. Understanding Theory of Change in international development. 2012.

  19. Jackson ET. Interrogating the Theory of Change: evaluating impact investing where it matters most. J Sustain Finance Investment. 2013;3(2):95–110. https://www.tandfonline.com/doi/abs/10.1080/20430795.2013.776257

  20. Engelbert M, Jain M, Bagai A, Parsekar SS. Improving routine childhood immunisation outcomes in low-income and middle-income countries: an evidence gap map. BMJ Open. 2022;12(11).

  21. Malik AA, Ahmed N, Shafiq M, Elharake JA, James E, Nyhan K, et al. Behavioral interventions for vaccination uptake: a systematic review and meta-analysis. Health Policy. 2023;137:104894. https://doi.org/10.1016/j.healthpol.2023.104894

  22. Coryn CLS, Noakes LA, Westine CD, Schröter DC. A systematic review of theory-driven evaluation practice from 1990 to 2009. Am J Evaluation. 2011;32:199–226.

  23. Craig P, Dieppe P, Macintyre S, Michie S, Nazareth I, Petticrew M. Developing and evaluating complex interventions: the new Medical Research Council guidance. BMJ. 2008;337(a1655):1–6. https://doi.org/10.1136/bmj.a1655

  24. Skivington K, Matthews L, Simpson SA, Craig P, Baird J, Blazeby JM, et al. A new framework for developing and evaluating complex interventions: update of Medical Research Council guidance. BMJ. 2021;374. https://www.bmj.com/content/374/bmj.n2061

  25. Carrillo B, Iglesias WJ, Trujillo JC. Attainments and limitations of an early childhood programme in Colombia. Health Policy Plan. 2015;30(7):906–16.

  26. Findley SE, Uwemedimo OT, Doctor HV, Green C, Adamu F, Afenyadu GY. Early results of an integrated maternal, newborn, and child health program, Northern Nigeria, 2009 to 2011. BMC Public Health. 2013;13(1):1–12. https://bmcpublichealth.biomedcentral.com/articles/10.1186/1471-2458-13-1034

  27. Goel S, Dogra V, Gupta SK, Lakshmi PVM, Varkey S, Pradhan N, et al. Effectiveness of Muskaan ek Abhiyan (the smile campaign) for strengthening routine immunization in Bihar, India. Indian Pediatr. 2012;49(2):103–8. https://link.springer.com/article/10.1007/s13312-012-0023-7

  28. Rahman M, Yunus FM, Shah R, Jhohura FT, Mistry SK, Quayyum T, et al. A controlled before-and-after perspective on the improving maternal, neonatal, and child survival program in rural Bangladesh: An impact analysis. PLoS One. 2016;11(9).

  29. Janssens W. Externalities in program evaluation: the impact of a women’s empowerment program on immunization. J Eur Econ Assoc. 2011;9(6):1082–113. https://doi.org/10.1111/j.1542-4774.2011.01041.x

  30. Nasir NM, Amran Y, Nakamura Y. Changing knowledge and practices of mothers on newborn care through mother class: an intervention study in Indonesia. J Trop Pediatr. 2017;63(6):440–6.

  31. Oche MO, Umar AS, Ibrahim MT, Sabitu K. An assessment of the impact of health education on maternal knowledge and practice of childhood immunization in Kware, Sokoto State. J Public Health Epidemiol. 2011;3(10):440–7. https://www.academicjournals.org/JPHE

  32. Okeke E, Glick P, Abubakar IS, Chari AV, Pitchforth E, Exley J, et al. Evaluating the midwives service scheme. 2017.

  33. Padilla A de J, Trujillo JC. An impact assessment of the child growth, development and care program in the caribbean region of Colombia. Cad Saude Publica. 2015;31(10):2099–109. https://www.scielo.br/j/csp/a/T9XMDbMctM7KP3Fsn3CLgLm/?lang=en

  34. Saggurti N, Atmavilas Y, Porwal A, Schooley J, Das R, Kande N, et al. Effect of health intervention integration within women’s self-help groups on collectivization and healthy practices around reproductive, maternal, neonatal and child health in rural India. PLoS One. 2018;13(8).

  35. Alatas V, Chandrasekhar AG, Mobius M, Olken BA, Paladines C, Alsan M, et al. When celebrities speak: a nationwide Twitter experiment promoting vaccination in Indonesia. 2019. https://www.nber.org/papers/w25589.ack

  36. Banwat ME, Lar L, Abok I, Yiltok E. Effect of peer education on knowledge, attitude and completeness of childhood routine immunization in a rural community of Plateau State. Res J of Health Sci. 2015;3(4).

  37. D’Souza VP, Umarani J. Teaching package improves mothers knowledge on vaccine preventable diseases and vaccination: a Quasi experimental study. Int J Res Med Sci. 2014;2(3):976.

  38. Gurley N, Shearer J, Srivastava Y, Mahapatra S, Desmond M. Impacts of community-led video education to increase vaccination coverage in Uttar Pradesh, India. 3ie Impact Evaluation Report 125, New Delhi: International Initiative for Impact Evaluation (3ie); 2020.

  39. Johri M, Chandra D, Kone KG, Sylvestre MP, Mathur AK, Harper S, Nandi A. Social and behavior change communication interventions delivered face-to-face and by a mobile phone to strengthen vaccination uptake and improve child health in rural India: randomized pilot study. JMIR Mhealth Uhealth. 2020;8(9):e20356. https://doi.org/10.2196/20356

  40. Memon ZA, Khan GN, Soofi SB, Baig IY, Bhutta ZA. Impact of a community-based perinatal and newborn preventive care package on perinatal and neonatal mortality in a remote mountainous district in Northern Pakistan. BMC Pregnancy Childbirth. 2015;15(1).

  41. Oyo-Ita A, Capblanch XB, Ross A, Hanlon P, Oku A, Esu E, et al. Impacts of engaging communities through traditional and religious leaders on vaccination coverage in Cross River State, Nigeria. New Delhi: International Initiative for Impact Evaluation (3ie); 2020.

  42. Pramanik S, Ghosh A, Goswami A, Das T, Nanda R, Forth P, et al. Impacts of the stimulate, appreciate, learn and transfer community engagement approach to increase immunization coverage in Assam, India. 3ie Impact Evaluation Report 130, New Delhi: International Initiative for Impact Evaluation (3ie); 2020.

  43. Powell-Jackson T, Fabbri C, Dutt V, Tougher S, Singh K. Effect and cost-effectiveness of educating mothers about childhood DPT vaccination on immunisation uptake, knowledge, and perceptions in Uttar Pradesh, India: a randomised controlled trial. PLoS Med. 2018;15(3).

  44. Usman HR, Akhtar S, Habib F, Jehan I. Redesigned immunization card and center-based education to reduce childhood immunization dropouts in urban Pakistan: a randomized controlled trial. Vaccine. 2009;27(3):467–72.

  45. Vaidyanathan R. Immunization coverage among under-five children living along a school student through child-to-child and child-to-parent information, education and communication strategy. Indian J Public Health. 2019;63(4):334–40.

  46. Younes L, Houweling TAJ, Azad K, Kuddus A, Shaha S, Haq B, et al. The effect of participatory women’s groups on infant feeding and child health knowledge, behaviour and outcomes in rural Bangladesh: a controlled before-and-after study. J Epidemiol Community Health. 2015;69(4):374–81. https://jech.bmj.com/content/69/4/374

  47. V Andrade CM, Viegas Andrade M, Chein F, Rodrigues de Souza L, Puig-Junoy J. Income transfer policies and the impacts on the immunization of children: the Bolsa Família Program Programas de transferência de renda e impactos na imunização das crianç as: o Programa Bolsa Família. Vol. 28. 2012.

  48. Andreoni J, Callen M, Khan MY, Jaffar K, Sprenger C, Best M, et al. Using preference estimates to customize incentives: an application to polio vaccination drives in Pakistan. J Eur Econ Assoc. 21(4):1428–77. https://ideas.repec.org/a/oup/jeurec/v21y2023i4p1428-1477..html

  49. Beck S, Pulkki-Brännström AM, San Sebastián M. Basic income – healthy outcome? Effects on health of an Indian basic income pilot project: a cluster randomised trial. J Dev Effect. 2015;7(1):111–26.

  50. Cahyadi N, Hanna R, Rizal BAO, Prima A, Satriawan E, Syamsulhakim E, et al. Cumulative impacts of conditional cash transfer programs: experimental evidence from Indonesia. Working Paper, 2018. https://www.nber.org/papers/w24670

  51. Carvalho N, Thacker N, Gupta SS, Salomon JA. More evidence on the impact of India’s conditional cash transfer program, Janani Suraksha Yojana: quasi-experimental evaluation of the effects on childhood immunization and other reproductive and child health outcomes. PLoS One. 2014;9(10). https://doi.org/10.1371/journal.pone.0109311

  52. Kusuma D, Thabrany H, Hidayat B, McConnell M, Berman P, Cohen J. New evidence on the impact of large-scale conditional cash transfers on child vaccination rates: the case of a clustered-randomized trial in Indonesia. World Dev. 2017;98:497–505.

  53. Okoli U, Morris L, Oshin A, Pate MA, Aigbe C, Muhammad A. Conditional cash transfer schemes in Nigeria: potential gains for maternal and child health service uptake in a national pilot programme. BMC Pregnancy Childbirth. 2014;14(1).

  54. Olken BA, Onishi J, Wong S. Should aid reward performance? Evidence from a field experiment on health and education in Indonesia. Am Econ J Appl Econ. 2014;6(4):1–34.

  55. Rahman MM, Pallikadavath S. Maternal and child health care services’ utilization data from the fourth round of district level household survey in India. Data Brief. 2019;23.

  56. Rahman MM, Pallikadavath S. How much do conditional cash transfers increase the utilization of maternal and child health care services? New evidence from Janani Suraksha Yojana in India. Econ Hum Biol. 2018;31:164–83.

  57. Rasella D, Aquino R, Santos CAT, Paes-Sousa R, Barreto ML. Effect of a conditional cash transfer programme on childhood mortality: a nationwide analysis of Brazilian municipalities. The Lancet. 2013;382(9886):57–64.

  58. Shei A, Costa F, Reis MG, Ko AI. The impact of Brazil’s Bolsa Família conditional cash transfer program on children’s health care utilization and health outcomes. BMC Int Health Hum Rights. 2014;14(1).

  59. Thomas RA. Essays of Ex ante evaluations of cash transfer programs. 2011.

  60. Wadhwa S, Foster A, Steinberg B. Conditional cash transfers and parental investment in daughters: evidence from India. 2019.

  61. Banerjee AV, Duflo E, Glennerster R, Kothari D. Improving immunisation coverage in rural India: clustered randomised controlled evaluation of immunisation campaigns with and without incentives. BMJ. 2010;340(7759):1291.

  62. Banerjee A, Chandrashekhar A, Duflo E, Dalpath S, Floretta J, Jackson M, et al. Evaluating the impact of interventions to improve full immunisation rates in Haryana, India. 3ie Impact Evaluation Report 126, New Delhi: International Initiative for Impact Evaluation (3ie); 2020.

  63. Chandir S, Khan AJ, Hussain H, Usman HR, Khowaja S, Halsey NA, et al. Effect of food coupon incentives on timely completion of DTP immunization series in children from a low-income area in Karachi, Pakistan: a longitudinal intervention study. Vaccine. 2010;28(19):3473–8.

  64. Demilew A, Girma M, McElwee E, Datta S, Barofsky J, Disasa T. Improving immunisation through supportive feedback and nonmonetary incentives: impact evaluation in Ethiopia. 2019.

  65. Seth R, Akinboyo I, Chhabra A. Mobile phone incentives for childhood immunizations in rural India. Pediatrics. 2018;141(4):20173455.

  66. Siddiqi DA, Ali RF, Munir M, Shah MT, Khan AJ, Chandir S. Effect of vaccine reminder and tracker bracelets on routine childhood immunization coverage and timeliness in urban Pakistan (2017-18): a randomized controlled trial. BMC Public Health. 2020;20(1):1–14. https://bmcpublichealth.biomedcentral.com/articles/10.1186/s12889-020-09088-4

  67. Hategeka C, Ruton H, Law MR. Effect of a community health worker mHealth monitoring system on uptake of maternal and newborn health services in Rwanda. Glob Health Res Policy. 2019;4(1).

  68. Brown VB, Oluwatosin OA, Ogundeji MO. Impact of training intervention on immunization providers’ knowledge and practice of routine immunization in Ibadan, south-western Nigeria: a primary health care experience. Pan Afr Med J. 2017;26:216.

  69. Dipeolu IO. Effect of mobile-phone reminder text messages on mother’s knowledge and completion of routine immunization in rural areas of Oyo State, Nigeria. 2017.

  70. Domek GJ, Contreras-Roldan IL, Bull S, O’Leary ST, Bolaños Ventura GA, Bronsert M, et al. Text message reminders to improve infant immunization in Guatemala: a randomized clinical trial. Vaccine. 2019;37(42):6192–200.

  71. Ekhaguere OA, Oluwafemi RO, Badejoko B, Oyeneyin LO, Butali A, Lowenthal ED, et al. Automated phone call and text reminders for childhood immunisations (PRIMM): a randomised controlled trial in Nigeria. BMJ Glob Health. 2019;4(2).

  72. Eze GU, Adeleye AO. Enhancing routine immunization performance using innovative technology in an urban area of Nigeria. West Afr J Med. 2015;34(1):1–8.

  73. Kazi AM, Ali M, Zubair K, Kalimuddin H, Kazi AN, Iqbal SP, et al. Effect of mobile phone text message reminders on routine immunization uptake in Pakistan: randomized controlled trial. JMIR Public Health Surveill. 2018;4(3).

  74. Modi D, Dholakia N, Gopalan R, Venkatraman S, Dave K, Shah S, et al. mHealth intervention “ImTeCHO” to improve delivery of maternal, neonatal, and child care services—a cluster-randomized trial in tribal areas of Gujarat, India. PLoS Med. 2019;16(10):e1002939. https://journals.plos.org/plosmedicine/article?id=10.1371/journal.pmed.1002939

  75. Murthy N, Chandrasekharan S, Prakash MP, Ganju A, Peter J, Kaonga N, et al. Effects of an mHealth voice message service (mMitra) on maternal health knowledge and practices of low-income women in India: findings from a pseudo-randomized controlled trial. BMC Public Health. 2020;20(1).

  76. Nagar R, Venkat P, Stone LD, Engel KA, Sadda P, Shahnawaz M. A cluster randomized trial to determine the effectiveness of a novel, digital pendant and voice reminder platform on increasing infant immunization adherence in rural Udaipur, India. Vaccine. 2018;36(44):6567–77.

  77. Nagar R, Ambiya MS, Singh P, Abdullah H, Banshiwal V, Stone L, et al. Impacts of a novel mHealth platform to track maternal and child health in Udaipur, India. 2020. https://www.3ieimpact.org/evidence-hub/publications/impact-evaluations/impacts-novel-mhealth-platform-track-maternal-and

  78. Owais A, Hanif B, Siddiqui AR, Agha A, Zaidi AK. Does improving maternal knowledge of vaccines impact infant immunization rates? A community-based randomized-controlled trial in Karachi, Pakistan. BMC Public Health. 2011;11.

  79. Uddin MJ, Shamsuzzaman M, Horng L, Labrique A, Vasudevan L, Zeller K, et al. Use of mobile phones for improving vaccination coverage among children living in rural hard-to-reach areas and urban streets of Bangladesh. Vaccine. 2016;34(2):276–83.

  80. Wallace AS, Peetosutan K, Untung A, Ricardo M, Yosephine P, Wannemuehler K, et al. Home-based records and vaccination appointment stickers as parental reminders to reduce vaccination dropout in Indonesia: a cluster-randomized controlled trial. Vaccine. 2019;37(45):6814–23.

  81. Carnell MA, Dougherty L, Pomeroy AM, Karim AM, Mekonnen YM, Mulligan BE. Effectiveness of scaling up the ‘three pillars’ approach to accelerating MDG 4 progress in Ethiopia. J Health Popul Nutr. 2014;32(4):549–63.

  82. Habib MA, Soofi S, Cousens S, Anwar S, Ul Haque N, Ahmed I, et al. Community engagement and integrated health and polio immunisation campaigns in conflict-affected areas of Pakistan: a cluster randomised controlled trial. Lancet Glob Health. 2017;5(6):e593-e603. https://www.thelancet.com/journals/lanpsy/article/PIIS2214-109X(17)30184-5/fulltext

  83. Mazumder S, Taneja S, Bahl R, Mohan P, Strand TA, Sommerfelt H, et al. Effect of implementation of Integrated Management of Neonatal and Childhood Illness programme on treatment seeking practices for morbidities in infants: cluster randomised trial. BMJ. 2014;349. https://doi.org/10.1136/bmj.g498

  84. Mohan P, Kishore B, Singh S, Bahl R, Puri A, Kumar R. Assessment of implementation of integrated management of neonatal and childhood illness in India. J Health Popul Nutr. 2011;29(6):629–38. https://www.banglajol.info/index.php/JHPN/article/view/9900

  85. Rao T. The impact of community health workers on childhood immunization: evidence from India–s ASHA worker program. 2013.

  86. Thomson DR, Amoroso C, Atwood S, Bonds MH, Rwabukwisi FC, Drobac P, et al. Impact of a health system strengthening intervention on maternal and child health outputs and outcomes in rural Rwanda 2005–2010. BMJ Glob Health. 2018;3(2):e000674. https://gh.bmj.com/content/3/2/e000674

  87. Varghese B, Roy R, Saha S, Roalkvam S. Fostering maternal and newborn care in India the Yashoda Way: does this improve maternal and newborn care practices during institutional delivery? PLoS One. 2014;9(1):e84145. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0084145

  88. Adamu AA, Uthman OA, Gadanya MA, Wiysonge CS. Implementation and evaluation of a collaborative quality improvement program to improve immunization rate and reduce missed opportunities for vaccination in primary health-care facilities: a time series study in Kano, Nigeria. Expert Rev Vaccines. 2019;18(9):969–91.

  89. Balasubramaniam S, Kumar S, Sethi R, Charurat E, Lalchandani K, Schuster A, et al. Quasi-experimental study of systematic screening for family planning services among postpartum women attending village health and nutrition days in Jharkhand, India. Int J Integr Care. 2018;18(1).

  90. Manyazewal T, Mekonnen A, Demelew T, Mengestu S, Abdu Y, Mammo D, et al. Improving immunization capacity in Ethiopia through continuous quality improvement interventions: a prospective quasi-experimental study. In: Infectious diseases of poverty. Vol. 7. BioMed Central Ltd; 2018.

  91. Basinga P, Gertler PJ, Binagwaho A, Soucat ALB, Sturdy JR, Vermeersch CMJ. Paying primary health care centers for performance in Rwanda. 2010. https://econ.worldbank.org

  92. Cristia J, Prado AG, Peluffo C. The impact of contracting in and contracting out basic health services: the guatemalan experience. World Dev. 2015;70:215–27.

  93. Cristia J, Evans WN, Kim B. Improving the health coverage of the rural poor: does contracting-out mobile medical teams work? J Develop Stud. 2015;51(3):247–61.

  94. Cristia JP, Evans WN, Kim B. Does contracting-out primary care services work? The case of rural Guatemala. 2011. https://www.iadb.org

  95. Pathak Y, Macours K. Women’s political reservation, early childhood development, and learning in India. 2017;65(4):741–66. https://www.journals.uchicago.edu/doi/abs/10.1086/692114

  96. Ryman TK, Trakroo A, Wallace A, Gupta SK, Wilkins K, Mehta P, et al. Implementation and evaluation of the Reaching Every District (RED) strategy in Assam, India, 2005-2008. Vaccine. 2011;29(14):2555–60.

  97. Sato R, Belel A. The effect of performance-based financing on child vaccinations in northern Nigeria. Vaccine. 2020;38(9):2209–15.

  98. Ngo DKL, Sherry TB, Bauhoff S. Health system changes under pay-forperformance: the effects of Rwanda’s national programme on facility inputs. Health Policy Plan. 2017;32(1):11–20.

  99. Talukder N, Rob U, Musa SAJM, Bajracharya A, Keya KT, Noor FR, et al. Evaluation of the impact of the voucher program for improving maternal health behavior and status in Bangladesh. Dhaka: Population Council; 2014.

  100. Helleringer S, Asuming PO, Abdelwahab J. The effect of mass vaccination campaigns against polio on the utilization of routine immunization services: a regression discontinuity design. Vaccine. 2016;34(33):3817–22.

  101. Mohanan M, Rajan VS, Swanson KM, Thirumurthy H. Information and facilitation interventions for accountability in health and nutrition: evidence from a randomized evaluation in India. Economic Research Initiatives at Duke (ERID) Working Paper No. 295, 2020.

  102. Wong BKC, Fadel SA, Awasthi S, Khera A, Kumar R, Menon G, et al. The impact of measles immunization campaigns in India using a nationally representative sample of 27,000 child deaths. eLife. 8:e43290.

  103. Busso M, Cristia J, Humpage S. Did you get your shots? Experimental evidence on the role of reminders. J Health Econ. 2015;44:226–37.

  104. Sengupta P, Benjamin AI, Myles PR, Babu BV. Evaluation of a community-based intervention to improve routine childhood vaccination uptake among migrants in urban slums of Ludhiana, India. J Public Health (Oxf). 2017;39(4):805–12.

  105. Atnafu A, Otto K, Herbst CH. The role of mHealth intervention on maternal and child health service delivery: findings from a randomized controlled field trial in rural Ethiopia. Mhealth. 2017;3:39–39.

  106. Prinja S, Nimesh R, Gupta A, Bahuguna P, Gupta M, Thakur JS. Impact of m-health application used by community health volunteers on improving utilisation of maternal, new-born and child health care services in a rural area of Uttar Pradesh, India. Tropical Medicine and International Health. 2017;22(7):895–907.

  107. Borkum E, Rangarajan A, Rotz D, Sridharan S, Sethi S, Manoranjini M. Evaluation of the team-based goals and performance based incentives (TBGI) intervention in Bihar Final Report. 2014.

  108. Bhuiya A, Hanifi SMA, Hoque S. Unlocking community capability through promotion of self-help for health: experience from Chakaria, Bangladesh. BMC Health Serv Res. 2016;16:105–17.

  109. Costa-Font J, Parmar D. Political agency and public healthcare: Evidence from India. WIDER Working Paper 2016/135. Helsinki: UNU-WIDER; 2016. https://doi.org/10.35188/UNU-WIDER/2016/179-6

  110. More NS, Bapat U, Das S, Alcock G, Patil S, Porel M, et al. Community mobilization in Mumbai slums to improve perinatal care and outcomes: a cluster randomized controlled trial. PLoS Med. 2012;9(7).

  111. More NS, Das S, Bapat U, Alcock G, Manjrekar S, Kamble V, et al. Community resource centres to improve the health of women and children in informal settlements in Mumbai: a cluster-randomised, controlled trial. Lancet Glob Health. 2017;5(3):e335–49.

  112. Sankar D. Improving early childhood development through community mobilization and integrated planning for children. 2013.

  113. Aggarwal S, Basurto P, Bulman G, Calvi R, Chakravorty U, Chaurey R, et al. The long road to health: healthcare utilization impacts of a road pavement policy in rural India. 2018.

  114. Banerjee R, Sachdeva A. Pathways to preventive health, evidence from India’s rural road program pathways to preventive health, evidence from India’s rural road program. USC-INET Research Paper No. 15-19, 2015. https://ssrn.com/abstract=2636999

  115. Chen YJ, Chindarkar N, Xiao Y. Effect of reliable electricity on health facilities, health information, and child and maternal health services utilization: evidence from rural Gujarat, India. J Health Popul Nutr. 2019;38(1):7.

  116. Weldemariam M. The impact of fiscal decentralization on education and health outcomes in Ethiopia: a regional panel data analysis. 2010.

  117. Oyo-Ita A, Bosch-Capblanch X, Ross A, Hanlon P, Oku A, Esu E, et al. Impacts of engaging communities through traditional and religious leaders on vaccination coverage in Cross River State, Nigeria. New Delhi: International Initiative for Impact Evaluation (3ie); 2020.

  118. Banerjee A, Chandrashekar A, Duflo E, Dalpath S, Floretta J, Jackson M, et al. Evaluating the impact of interventions to improve full immunisation rates in Haryana. 2020.

  119. Carmichael SL, Mehta K, Raheel H, Srikantiah S, Chaudhuri I, Trehan S, et al. Effects of team-based goals and non-monetary incentives on front-line health worker performance and maternal health behaviours: A cluster randomised controlled trial in Bihar, India. BMJ Glob Health. 2019;4(4).

  120. Abayneh S, Lempp H, Alem A, Kohrt BA, Fekadu A, Hanlon C. Developing a Theory of Change model of service user and caregiver involvement in mental health system strengthening in primary health care in rural Ethiopia. Int J Ment Health Syst. 2020;14(1):1–17. https://ijmhs.biomedcentral.com/articles/10.1186/s13033-020-00383-6

  121. Aromatario O, Van Hoye A, Vuillemin A, Foucaut AM, Pommier J, Cambon L. Using Theory of Change to develop an intervention theory for designing and evaluating behavior change SDApps for healthy eating and physical exercise: the OCAPREV theory. BMC Public Health. 2019;19(1).

  122. Paina L, Peters DH. Understanding pathways for scaling up health services through the lens of complex adaptive systems. Health Policy Plan. 2012;27:365–73. https://academic.oup.com/heapol/article/27/5/365/751682

  123. Chaudhuri A, Biswas N, Kumar S, Jyothi A, Gopinath R, Mor N, et al. A Theory of Change roadmap for universal health coverage in India. Front Public Health. 2022;10:1040913.

  124. Babatunde GB, van Rensburg AJ, Bhana A, Petersen I. Identifying multilevel and multisectoral strategies to develop a Theory of Change for improving child and adolescent mental health services in a case-study district in South Africa. Child Adolesc Psychiatry Ment Health. 2022;16(1):1–19. https://capmh.biomedcentral.com/articles/10.1186/s13034-022-00484-9

  125. Meiksin R, Melendez-Torres GJ, Falconer J, Witzel TC, Weatherburn P, Bonell C. Theories of change for e-health interventions targeting HIV/STIs and sexual risk, substance use and mental ill health amongst men who have sex with men: systematic review and synthesis. Syst Rev. 2021;10(1):1–28. https://systematicreviewsjournal.biomedcentral.com/articles/10.1186/s13643-020-01523-2

  126. Brown Urban J, Linver MR, Kornak-Bozza Y, Macdonnell M, Buckley J. Combining STEM and character development in an out-of-school time program: participatory practices for developing and validating a Theory of Change. J STEM  Educ. 2023;23(4). https://www.jstem.org/jstem/index.php/JSTEM/article/view/2535

  127. Gutiérrez-Barreto SE, Sosa-Tinoco E, Rojas-Calixto O, Deniss-Navarro Z, Avila-Avila A, Gutierrez JP. Evaluating the design of the integrated care for older people: a theory of change approach. Front Med (Lausanne). 2023;10:1166196.

  128. Jones B, Paterson A, English M, Nagraj S. Improving child health service interventions through a Theory of Change: a scoping review. Front Pediatr. 2023;11:1037890.

  129. Breuer E, Lee L, De Silva M, Lund C. Using theory of change to design and evaluate public health interventions: a systematic review. Implement Sci. 2016;11(1).

  130. Bertone MP, Palmer N, Kruja K, Witter S. How do we design and evaluate health system strengthening? Collaborative development of a set of health system process goals. Int J Health Plann Manage. 2023;38(2):279–88.

  131. Adam T. Advancing the application of systems thinking in health. Health research policy and systems. Vol. 12. BioMed Central Ltd.; 2014.

  132. Witter S, Palmer N, Balabanova D, Mounier-Jack S, Martineau T, Klicpera A, et al. Health system strengthening—reflections on its meaning, assessment, and our state of knowledge. Int J Health Plann Manage. 2019;34(4):e1980–9.

  133. Moore GF, Evans RE. What theory, for whom and in which context? Reflections on the application of theory in the development and evaluation of complex population health interventions. SSM Population Health. 2017;3:132–35. https://doi.org/10.1016/j.ssmph.2016.12.005

  134. Funnell SC, Rogers PJ. Purposeful program theory: effective use of theories of change and logic models. 2011;550. https://books.google.com/books/about/Purposeful_Program_Theory.html?id=A9Iid1tcGwgC

  135. Aqil A, Silvestre E, Hotchkiss D, Maniscalco L. Health systems strengthening: monitoring, evaluation, and learning guide. North Carolina, Chapel Hill: MEASURE Evaluation; 2017.

  136. UNICEF. Implementation research for immunization. New York: UNICEF; 2015.

  137. Greene SM, Reid RJ, Larson EB. Implementing the learning health system: from concept to action. Ann Intern Med. 2012;157(3):207–10.

  138. White H. Theory-based impact evaluation: principles and practice. 3ie Working Paper 3, International Initiative for Impact Evaluation; 2009. https://3ieimpact.org/evidence-hub/publications/working-papers/theory-based-impact-evaluation-principles-and-practice

Download references

Acknowledgements

We would like to thank Prof. Maren Duvendack, University of East Anglia, United Kingdom, for her inputs. We would also like to thank Sebastian Martinez at 3ie for his overall guidance and support for the review.

Funding

This review was supported by funding from the Bill and Melinda Gates Foundation (BMGF) (INV008461).

Author information

Authors and Affiliations

  1. Impact Evaluation Initiative (3ie), 3ie, Unit no. 306, 3rd Floor, Rectangle-1, D-4, Saket District Centre, New Delhi, 110017, India

    Lalitha Vadrevu, Monica Jain & Shradha S. Parsekar

Authors
  1. Lalitha Vadrevu
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Monica Jain
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Shradha S. Parsekar
    View author publications

    You can also search for this author in PubMed Google Scholar

Contributions

LV conducted the data extraction of the studies included in the review. LV was responsible for the analysis and drafting of the first draft. MJ and SP finalized the manuscript.

Corresponding author

Correspondence to Lalitha Vadrevu.

Ethics declarations

Ethics approval and consent to participate

Not applicable.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Supplementary Material 2

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Vadrevu, L., Jain, M. & Parsekar, S.S. Analyzing the usage of theories of change for routine immunization programs -- a review of impact evaluations from LMICs. J Health Popul Nutr 43, 141 (2024). https://doi.org/10.1186/s41043-024-00615-2

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s41043-024-00615-2

Keywords

Journal of Health, Population and Nutrition

ISSN: 2072-1315

Contact us