Table 3 Evaluation study design and data collection methods of the included studies

Aveling, E. L. (2010). “The impact of aid chains: Relations of dependence or supportive partnerships for community-led responses to HIV/AIDS?” AIDS Care Psychological and Socio-Medical Aspects of AIDS/HIV 22(SUPPL. 2): 1588–1597.

The Happy Couples Programme (HCP). “The HCP brings into ‘partnership’ international and local development organisations, government institutions and military families. The programme is an HIV/AIDS prevention programme using peer education with Cambodian soldiers and their wives, which aims to increase reproductive health knowledge, condom use and access to health services”. (p. 1589)

The evaluation, “…examines how aid relationships can both promote and undermine the possibility of successful community-led responses” to HIV/AIDS. (p. 1589)

Case study ethnography. Data collection methods included semi-structured, qualitative interviews review of documentary evidence, observations of programme activities.

Bayley, O., H. Chapota, E. Kainja, T. Phiri, C. Gondwe, C. King, B. Nambiar, C. Mwansambo, P. Kazembe, A. Costello, M. Rosato and T. Colbourn (2015). “Community-linked maternal death review (CLMDR) to measure and prevent maternal mortality: a pilot study in rural Malawi.” BMJ Open 5(4): e007753.

“The process was triggered in the event of any maternal death, by community CLMDR team members hearing about a death in their area. Stage 1 began with the woman’s family giving consent for the process, followed by a verbal autopsy, or structured interview, including multiple open-ended free-text questions about the events leading up to her death. This form was used to record data at all stages of the process and designed to facilitate discussion and communication between participants.

Stage 2 was a meeting held in the woman’s local area by the community team. They recorded factors they believed contributed to the woman’s death, and suggested strategies to prevent future deaths.

Stage 3 was a meeting held at the woman’s local health facility or at the district hospital dependent on where the death occurred, with a broad spectrum of health center staff, district hospital staff and the HSA. The HSA reported the information from the verbal autopsy and the community team discussions. Participants agreed on a medical cause of death and health facility factors that may have contributed to the death, after which they recorded the strategies that they planned to prevent future deaths. Action points were assigned to individual health center and district hospital staff to implement. Stage 4 was a public meeting held in the woman’s local community, attended by district hospital and health center representatives, the HSA, community leaders, and community members—all were welcome to attend. The HSA sought the family’s consent to summarize the case in order to facilitate an open discussion of all relevant factors. The health workers presented their planned action points. The community agreed on community factors that may have contributed to the death and planned their own strategies, assigning action points for individuals to implement. Stage 5 was a bimonthly meeting, which provided an opportunity for community and health facility representatives to hear about progress on implementing action points, celebrate successes, and to identify and overcome any barriers to action. An additional meeting of traditional leaders was held quarterly in order to share innovations and lessons learned across the whole district.” (p. 3–4)

"Our study describes the Malawian context and identifies six weaknesses of the current MDR system. We present the pilot study of the CLMDR process over a 1-year period, and the results of how it can overcome these weaknesses and provide an estimate of maternal mortality. We conclude with thoughts on the added value and applicability of the CLMDR approach". (p. 2)

No specific evaluation design provided. The paper discusses the process of conducting death reviews in the community, i.e., the design of the intervention rather than the evaluation.

Björkman M, Svensson J. Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda. The Quarterly Journal of Economics. 2009;124(2):735-69.

Community mobilization through participatory surveys that solicit user feedback on the performance of public services against set standards “…aimed at enhancing community involvement and monitoring in the delivery of primary health care…” (p. 739)

“To examine whether community-based monitoring works, we designed and conducted a randomized field experiment in fifty communities from nine districts in Uganda”. (p. 736)

"First, data were required to assess how the community at large views the quality and efficacy of service delivery. We also wanted to contrast the citizens’ view with that of the health workers. Second, data were required to evaluate impact." (p. 740)

“Two surveys were implemented: a survey of the fifty providers and a survey of users. Both surveys were implemented prior to the intervention (data from these surveys formed the basis for the intervention) and one year after the project had been initiated” (p. 740–741)

Blake, C., N. A. Annorbah-Sarpei, C. Bailey, Y. Ismaila, S. Deganus, S. Bosomprah, F. Galli and S. Clark (2016). “Scorecards and social accountability for improved maternal and newborn health services: A pilot in the Ashanti and Volta regions of Ghana.” International Journal of Gynecology and Obstetrics 135(3): 372–379.

“The initiative was designed to strengthen partnerships between clients, providers, and the community at large for improved maternal and newborn health (MNH) care through a social accountability process using scorecards. Before carrying out scorecard assessments, health providers and community-based NGOs were trained on MNH rights and client care to ensure a common understanding of entitlements in MNH service delivery. Although this intervention did not focus on clinical skills building for quality EmONC, the aim was to improve the enabling environment for EmONC and engage the community at large in this endeavor.” (p. 373)

The aims were to, “…examine qualitative and quantitative evidence from the social accountability intervention used by Evidence 4 Action to assess the effectiveness of engaging multiple health and non-health sector stakeholders to improve MNH services at facility level. It also identifies some limitations to this strategy and makes recommendations for future interventions of a similar nature.” (p.373)

“The study had two components. The quantitative component comprised two rounds of facility assessments. The qualitative component prospectively assessed the impact of changes in policy, attitudes, and/ or practices.” (p. 373)

“An independent prospective policy study carried out by external researchers followed the E4A program with the aim of understanding the resulting changes at district and regional level. Data collection focused on process tracing to assess whether and how the scorecard process contributed to changes in policies or to changes in attitudes or practices among key stakeholders. This data collection included regular meeting observations and analysis of documents, as well as repeat interviews with a broad base of MNH actors, including local government staff, district assembly members, health facility managers, community leaders, and organizations.” (p. 374) [see Nove et al for more information about Evidence 4 Action]

Boydell, V., S. Neema, K. Wright and K. Hardee (2018). “Closing the Gap between People and Programs: Lessons from Implementation of Social Accountability for Family Planning and Reproductive Health in Uganda.” African Journal of Reproductive Health 22(1): 73–84.

“Between 2008 and 2013, the German Foundation for World Population (DSW) and Reproductive Health Uganda (RHU) implemented the European Commission-funded HAP in five districts in Uganda. HAP was a social accountability project aimed at empowering civil society and citizens in Uganda to effectively participate in policy priorities, monitoring their implementation and holding duty bearers to account for their promises. The project outlined a theory of change (TOC) which informed the project‘s actions. First the capacity of civil society organizations (CSOs) for advocacy, resource mobilization and civic education was built. With this improved capacity, the CSOs then could mobilize communities to participate in monitoring services and actively interact with decision-makers to generate solutions and work in partnership to bring about the desired changes. The project started with selecting national CSOs for a series of capacity building activities. The trained CSOs received a sub-award to support community activities such as (1) increasing community members‘ awareness about family planning and their entitlements; (2) building civil society and community coalitions at the district level; (3) undertaking civic education with local communities; and (4) training community members on holding dialogues with health care providers and health officials to jointly identify challenges, priorities and solutions.” (p. 75)

The paper assessed "results of retrospective implementation research into a five-year social accountability project in Uganda that focused on family planning and reproductive health. A mix of methods was used examine the project‘s implementation in three districts in Uganda between 2009 to 2013” (p.73) “To address these gaps this paper presents findings from an implementation study of a social accountability project focused on FP and reproductive health (RH), the Healthy Action Project (HAP), implemented in Uganda between 2009 and 2013. This paper examines the implementation of HAP with a focus on family planning.” (p. 74)

Retrospective implementation research using document review, Political Economy Analysis, and interviews.

Dasgupta, J. (2011). “Ten years of negotiating rights around maternal health in Uttar Pradesh, India.” BMC International Health and Human Rights 11 (SUPPL. 3) (S4).

“This paper reviews documents of the last ten years describing the experiences of a Non-Governmental Organization, SAHAYOG, in working with a civil society platform, the Healthwatch Forum, to develop ‘rights based’ strategies around maternal health. The paper builds an analysis using recent frameworks on accountability and gendered rights claiming to examine these experiences and draw out lessons regarding rights claiming strategies for poor women.” (from abstract)

“This paper interrogates the process of civil society action around maternal mortality in Uttar Pradesh to ask why the issue of maternal deaths never becomes a ‘political’ issue, why the agent of accountability is never clear and despite some gains at the localized sites, overall why the health system and bureaucracy remain inert; and what needs to be done differently.” (p.4)

Reflective, qualitative study. The paper uses “organizational records, including unpublished internal and external evaluation reports, in-house publications and web-based documents describing the experiences of SAHAYOG’s work of the last ten years […and…] unpublished reports of community based participatory approaches […], and several rounds of ‘policy dialogues’.” (p.4) “Armed with information about their entitlements and state provisions, the MSAM women, in an exercise of ‘active citizenship’ through monitoring and advocacy, took up various aspects of the NRHM each year for interrogation. At the start they revealed local corruption in the appointment of the ASHA workers (2006), then they audited the payment of the conditional cash transfer under the JSY (2007-8); they examined how ‘untied’ health budgets are spent locally and how much poor families are spending (2009), and recently they audited the compliance of health sub-centres with the Indian Public Health Standards (2010). Every NRHM monitoring exercise was followed by a formal presentation to the district health officials by the NGOs and MSAM women, as well as presentations at the state capital, Lucknow, usually in the presence of state officials and the media.” (p. 7)

Dasgupta, J., Y. K. Sandhya, S. Lobis, P. Verma and M. Schaaf (2015). “Using Technology to Claim Rights to Free Maternal Health Care: Lessons about Impact from the My Health, My Voice Pilot Project in India.” Health Hum Rights 17(2): 135–147.

“The Mera Swasthya, Meri Aawaz pilot project was developed to test whether a free telephone hotline connected to Ushahidi (www.ushahidi.com)—an open-source data management system that aggregates and displays data—could be tailored for illiterate women and used to monitor demands for informal payments. The implementers also sought to understand how the project could inform and strengthen grassroots advocacy efforts around maternal health, how it could affect women’s ability to claim their rights to maternal health care, and whether scale-up was feasible. To that end, it documented factors that contributed to success and failure, the project’s adaptation over time, challenges, and remaining questions.” (p.138) “The system works as follows: Women call the toll-free hotline to report having been asked to pay informal payments at a hospital. Each hospital in the project’s districts is assigned a four-digit code. Callers are asked to enter the hospital’s four-digit code as well as additional codes corresponding to the amount and purported justification for the payment (for example, “Press 2 if money was requested to pay for drugs.”) The information collected is then mapped in an Ushahidi installation and can be viewed at www.meraswasthyameriaawaz.org. Callers reporting emergencies are immediately routed to a live person; the emergency line is staffed 24 hours a day by a representative of the partnering community-based organizations.” (p. 139)

To assess the acceptability and effectiveness of the pilot project.

Qualitative mixed methods plus quantitative records of reports to the hotline (causality not directly assessed, a limitation the authors acknowledge)

“By the end of the project, MSAM members were known for challenging informal payments. Therefore, some primary health clinic staff stopped demanding informal payments once they knew that the woman was in some way affiliated with MSAM. The staff tended to treat women better in such cases” (p.142) “One example comes from Azamgarh. Through focus group discussions with MSAM women, we learned that following a block-level sharing of the Mera Swasthya, Meri Aawaz data, government officials took immediate action to remedy problems identified at one facility, including by fixing the water supply, improving electricity, providing free medicines, and offering food to women in the hospital following delivery. In addition, staff behavior toward women improved. The additional director of the Azamgarh District stated that the act of registering complaints was very important and that the Ushahidi data was useful because it made officials realize the enormity of the problem. Our analysis of the reporting patterns showed that the number of reports made about this particular facility dropped from an average of 18 reports per month before the block-level dialogue (January to November 2012) to 3 reports per month after the dialogue. The comments of the additional director and others lead us to believe that this decrease in reports was likely because requests for informal payments decreased. In this case, the dialogue was a catalytic event, as it triggered positive changes that included not only reductions in demands for informal payments but also improvements in staff behavior and infrastructure. The success of this dialogue also provided a boost to the MSAM women’s confidence in their ability to effect improvements in their health facilities.” (p. 143)

de Souza, R. (2009). “Creating ‘communicative spaces’: a case of NGO community organizing for HIV/AIDS prevention.” Health Communication 24(8): 692–702.

“The goal of the project is to reduce the prevalence of HIV/AIDS among female sex workers through participatory and empowering process of information dissemination and capacity building…The project will enable the sex worker community to participate in planning, monitoring, and evaluating interventions that affect their lives” (Project proposal, quoted in methods section, p. 694)

“What are the processes NGOs use to organize marginalized communities for HIV/AIDS prevention?” (p. 694) The study “seeks to understand ways in which the ideal of civil society can be achieved through more “civil” practices […] a case wherein an NGO uses practices within the discursive sphere to resurrect community voices.” (p. 694)

This research on an existing programme used a case study approach; “the case study method allows for data to be collected in a variety of ways (e.g., interviews, focus groups, and documents), thereby allowing for these complex processes to be identified. Finally, the case study method was used because, as with other ethnographic methods, it allows for attention to be given to the sociocultural context within which behaviors occur, an important feature of the culture-centered approach.” (p. 694)

George, A. S., D. Mohan, J. Gupta, A. E. LeFevre, S. Balakrishnan, R. Ved and R. Khanna (2018). “Can community action improve equity for maternal health and how does it do so? Research findings from Gujarat, India.” Int J Equity Health 17(1): 125.

Raising awareness, community monitoring, and dialogue with government health providers and authorities including use of report cards. (p. 9)

The study assesses the effects of community action on access to facility deliveries by marginalized groups across public and private sectors. It also evaluates the implementation processes that underpin community action and accountability for maternal health in Gujarat, India. (p. 2)

Document review, interviews, non-probability sample questionnaire survey. “The study combined qualitative data (project documents and 56 stakeholder interviews thematically analyzed) with quantitative data (2395 women's self-reported receipt of information on entitlements and use of services over 3 years of implementation monitored prospectively through household visits). Multivariable logistic regression examined delivery care seeking and equity.” (from Abstract). “The data are not based on any pre-determined sampling design and represent the efforts of volunteers to gather data from as many women as possible” (p. 4)

Gullo S, Galavotti C, Kuhlmann AS, Msiska T, Hastings P, Marti CN. Effects of a social accountability approach, CARE's Community Score Card, on reproductive health-related outcomes in Malawi: A cluster-randomized controlled evaluation. PLoS ONE. 2017;12 (2), e0171316.

Scorecards (see below)

“To this end, we designed a cluster-randomized control evaluation to assess the effectiveness of CARE’s Community Score Card (CSC) [11], a social accountability approach, to improve reproductive health-related outcomes in Ntcheu, Malawi”. (p. 2)

“CARE’s CSC was assessed in a cluster-randomized trial in the catchment areas of20 health facilities the in Ntcheu district of Malawi.” (p. 3)

See Gullo et al. 2018 for further details.

Gullo, S., A. S. Kuhlmann, C. Galavotti, T. Msiska, C. Nathan Marti and P. Hastings (2018). “Creating spaces for dialogue: a cluster-randomized evaluation of CARE's Community Score Card on health governance outcomes.” BMC Health Services Research 18(1): 858.

“The CSC [community score card] intervention consists of five phases… planning and preparation… involves identifying the sectoral and geographic scope of the initiative, understanding the context and barriers both service providers and users face, training facilitators, and securing cooperation and buy-in from all participating parties, including government officials. In phase 2, the CSC is conducted with the community via focus group discussions with community members (separated into groups such as men, women, youth, etc.) to identify and prioritize issues they are facing in accessing services. Identified issues are organized into themes and a measureable indicator is developed for each theme. The indicators are then verified and scored by the community, generating a Score Card. The community also indicates reasons for why a particular score was given and creates suggestions for improvement. The same process of issue generation and indicator development is conducted with service providers in phase 3; through focus group discussions, service providers identify issues they are facing in delivering quality services, develop and score indicators, give reasons for the scores, and make suggestions for improvement. Phase 3 can occur either after or concurrently with phase 2.

The CSC comes to life in phase 4 at the interface meeting, during which community members and service providers are joined by local government officials and other power holders to share and discuss their respective Score Cards, issues and priorities. This joint conversation gives way to locally identified solutions and a community-wide action plan for service improvement. Finally, phase 5 involves action plan implementation, monitoring, and evaluation in which community members, service providers, government staff and additional power-holders all have a role to play in reviewing and monitoring progress on indicators. This cycle is repeated (minus the initial planning and preparation stage) every six months: communities and service providers reconvene to discuss issues (and generate new ones, if needed), re-score the indicators and discuss reasons for changes, and then meet in an interface meeting to review their respective Score Cards, in an on-going cycle of problem identification, solution generation, implementation of improvements, and mutual accountability.” (p. 4)

Evaluate programme: “CARE’s Community Score Card (CSC), a social accountability approach, to improve reproductive health-related outcomes in Ntcheu, Malawi. The theory of change underlying the CSC intervention suggests that bringing together community members, health workers, and local officials to a) identify barriers and facilitators of service use and delivery, b) prioritize actions, and c) jointly monitor improvements will result in new and expanded spaces for inclusive, effective dialogue and negotiation. This, in turn, will empower both women and health workers in the community, leading to improved health behaviors, increased service utilization, and higher quality and more equitable service delivery. Ultimately, these changes, along with system and institutional changes, should decrease maternal and neonatal mortality in communities. Therefore, this evaluation aims to test the effectiveness of CARE’s CSC on maternal and reproductive health-related outcomes.” (p. 2)

“CARE’s CSC was assessed in a cluster-randomized trial in the catchment areas of 20 health facilities the in Ntcheu district of Malawi. Health facilities were matched in pairs and one facility from each pair was randomly assigned to participate in the CSC and the other was assigned as a control health facility. Two-stage cluster sampling was used to select group villages and villages for participation in the study.” (p.3) “We used data obtained from government census, district, and local office sources to construct the population from which we would draw the sample. Among intervention health facilities, there were 56 group villages (GVs) that contained 290 villages with a total population of 228,029. Among control health facilities, there were 36 GVs that contained 228 villages with a total population of 170,201. Using UNICEF’s probability-proportional-to-size (PPS) sampling method, we selected twenty GVs (i.e., clusters) from the intervention area and twenty GVs in the control area to serve as the primary sampling units. One of the largest intervention health facilities contained eight sampled GVs. Because we could not feasibly implement the CSC for all eight GVs in a single health facility, four GVs were dropped (leaving 16 GVs in the sample), and the PPS sample for this health facility was obtained from the remaining four GVs. The CARE Malawi team purposively identified 64 villages from the 16 intervention GVs in which to work, and randomly selected 64 villages in the 20 control GVs; the same PPS method described above was used to select villages. The number of individuals sampled in each village was determined by number of eligible women in a village multiplied by the sampling proportion for the condition (i.e., the required sample size divided by the total eligible population). We sized the sample to detect a 10% change in institutional births, based on the prevailing rates of institutional births in Ntcheu (78%), prior to baseline. Given the hypothesized effect size, our power analysis determined a sample of 650 women per treatment condition (power = .80, 2-tailed α = .05, non-response = 5%, and design effect = 2.0).” (p. 6)

“Community members and service providers developed 12 indicators to track progress, for example, reception of clients at the facility, level of male involvement in maternal newborn health (MNH) issues, and availability of transportation for referrals during labor and delivery. CSC participants and service providers generated similar issues, but from their different perspectives. For example, ‘relationship with providers’ was an indicator for both: from the community side this referred to how providers treated them, whereas from the provider’s side, it referred to things like patients not listening to them, or following their guidance. The service providers also generated one additional indicator—availability of supervisory support—for a total of 13 Score Card indicators. In an open discussion, participants agreed on scores for each indicator using a scale from 0–100. This was done with the communities and the service providers separately, and then, during the interface meeting the Score Cards were discussed and actions to improve scores were agreed upon. For each intervention site, there were 1–4 community Score Cards and 1 service provider Score Card. The same indicators were used across all 10 intervention sites and were re-scored during each 6 month CSC cycle.” (pp.8–9)

Hamal, M., T. de Cock Buning, V. De Brouwere, A. Bardaji and M. Dieleman (2018). “How does social accountability contribute to better maternal health outcomes? A qualitative study on perceived changes with government and civil society actors in Gujarat, India.” BMC Health Serv Res 18(1): 653.

“The Government of Gujarat has taken several initiatives to improve maternal health services, such as the Chiranjeevi Yojana (free childbirth services for poor women in private health facilities), the Kasturba Poshan Sahay Yojana (financial assistance for poor pregnant women).” (p. 2)

“This study explores the existing social accountability mechanisms for maternal health, the factors they address and how the results of these mechanisms are perceived.” (From abstract)

Qualitative research using in depth interviews and focus group discussions with individuals from civil society and the government health system (N = 10, and with clients/beneficiaries (N = 26)). (p. 4)

Hanson C, Waiswa P, Marchant T, Marx M, Manzi F, Mbaruku G, et al. Expanded Quality Management Using Information Power (EQUIP): protocol for a quasi-experimental study to improve maternal and newborn health in Tanzania and Uganda (2014). Implementation Science 9(1):41.

Quality management: “QM involves applying a set of principles to improve quality: conceptualizing work as processes (e.g., following a case-management guideline), designing processes to reduce errors, focusing improvement efforts on the most vital processes, satisfying both clients and employees, monitoring quality, using scientific and statistical thinking, creating new organizational structures (e.g., quality improvement teams), and involving all workers in quality improvement. QM also includes a structured problem-solving methodology, which uses teams to improve quality with continuous plan-do-study-act (PDSA) cycles, which monitor indicators, identify problems, understand causes, implement solutions, check if solutions are working, and modify solutions as needed.” (p. 2)

“EQUIP aims to test whether a Quality Management (QM) approach at three levels of care and supported by district level report cards generated by continuous surveys can improve the quality and utilization of services for mothers and newborns”. (p. 8) “The specific objectives are: 1. To assess the effects of the EQUIP intervention on uptake and quality of care of key maternal and newborn health interventions;

2. To assess the feasibility and acceptability of the intervention;

3. To model the potential impact of the intervention using the Lives Saved Tool (LiST);

4. To estimate cost and cost-effectiveness of the intervention.” (p. 3)

[NB this is a protocol] “The evaluation compares intervention and comparison districts with respect to change in utilization and quality of healthcare using indicators of coverage, service quality and knowledge”. (p. 7) “During the entire study period, ongoing data collection via continuous, high quality household and heath facility surveys is used to estimate pre- and post-intervention outcome levels in one intervention and one non-randomly selected comparison district each in Uganda and Tanzania. The continuous household surveys and health facility censuses cover implementation and comparison districts. The QM intervention, supported by report cards using data generated by the continuous surveys, is implemented in intervention districts only. For evaluation, changes over time in quality and uptake of key maternal and newborn interventions in intervention areas are compared with changes over time in comparison areas, with careful attention paid to contextual factors that also vary over time” (p. 3) “A qualitative sub-study on feasibility and acceptability includes: how, when, and with what intensity the intervention is implemented in the intervention district; how the intervention worked at different levels; and changes and observations reported by QITs. In-depth interviews with district staff involved in the project are used to assess the acceptability of the QM approach and feasibility of implementation within the district structure. The evaluation uses a non-interrupted time-series approach to compare changes over time in primary outcomes (see below) in intervention and comparison areas. We generate a single estimate of effect for each primary outcome, adjusting for confounding factors and baseline levels. Provided that utilization, quality or coverage improves sufficiently for an effect on survival to be plausible, we will also use the Lives Saved Tool (LiST) to model the potential impact of the intervention on child survival” (p. 7)

Hulton, L., Z. Matthews, A. Martin-Hilber, R. Adanu, C. Ferla, A. Getachew, C. Makwenda, B. Segun and M. Yilla (2014). “Using evidence to drive action: A ‘revolution in accountability’ to implement quality care for better maternal and newborn health in Africa.’ Int J Gynaecol Obstet 127(1): 96-101.

"In 2011, the Commission on Information and Accountability (CoIA) for Women’s and Children’s Health published a report with 10 key recommendations […] Better use of information to improve results, better tracking of resources, and better oversight of results and resources globally and nationally were the three areas within which the CoIA recommended action. Making countries and stakeholders accountable for women’s and children’s health was a central tenet of the recommended actions. […] Evidence for Action (E4A)—a DFID-funded program that contributes to accountability in MNH—was launched soon after the publication of the Commission’s Final Report and was designed explicitly as a contribution to delivering the CoIA framework in six African countries: Ethiopia, Ghana, Malawi, Nigeria, Sierra Leone, and Tanzania. […] the program seeks to address resource deficiencies in these countries that contribute to suboptimal quality of MNH care. A strategic combination of evidence and advocacy to stimulate and strengthen accountability underpins this program.” (p. 96)

“The E4A program therefore invests in catalytic interventions that stimulate accountability and involve the use of well-communicated evidence: first via better use of evidence by decision-makers and second via better communicated information as a basis for influencing public perceptions and political action.” (p. 97)

“Data capture through scorecards, dashboards, and maternal death reviews (MDRs) are some of the ways in which E4A has been supporting evidence building and decision-making. For example, marshalling existing data into district dashboards (a visual display of key MNH indicators and other health resources consolidated on a single screen in the form of tables and graphs) that are communicated to and increasingly owned by district decision-makers has exposed the variation across districts and among facilities in Malawi, and stimulated discussions among them to make informed decisions about planning and resource allocation locally.” (p. 97)

“We aim to address the specific need for more “country-level detail about how the Commission’s recommendations are leading to change and action” as identified in the recent iERG report” (p. 97). “In addition to forming the basis for program design, E4A’s theory of change also provides us the scope to test whether data and evidence drive improvements through advocacy and social accountability.” (p. 99)

This is high level reporting of results with minimal methods description. Methods are reported in Nove et al. (see separate entry)

Kamuzora, P., S. Maluka, B. Ndawi, J. Byskov and A. K. Hurtig (2013). “Promoting community participation in priority setting in district health systems: experiences from Mbarali district, Tanzania.” Global Health Action 6: 22669.

“REACT was a 5-year project aimed at testing the application and effects of Accountability for Reasonableness (AFR) approach to priority setting in resource-constrained settings. AFR is a comprehensive framework which provides structure for stakeholders to establish priorities for their specific contexts, while taking into account limited resources and regulatory conditions. The REACT project aimed at implementing the four conditions of the AFR framework (see Table 3).” (p. 4)

“The aim of this article is to provide the experience of implementing community participation and the challenges of promoting it in the context of resource-poor settings, weak organizations, and fragile democratic institutions.” (p. 1)

Before and after design, with mixed qualitative data collection methods.

“This article is based on two major sources of data: analysis of documents and key informant interviews. Documents analyzed included minutes of the ART, CHMT, and annual planning and priority-setting reports. Key informant interviews were conducted with various stakeholders in the district and region. Furthermore, all six representatives of the marginalized groups, namely women, youth, elderly, disabled, and people living with HIV/AIDS, who joined the CHMT for priority setting and budget discussion were interviewed. Interviews were conducted in two phases. Twenty-one interviews were carried out with various stakeholders in the district toward the end of the REACT project in August 2010. An additional 14 interviews were carried out 1 year after the end of the project in April 2012 by the researcher (S. M.) who was not directly involved in the implementation of the project in the district. In the second phase, respondents included only those who were directly involved in the priority setting and budget discussions namely CHMT members and representatives of the communities. In total, 35 interviews were carried out and analyzed” (p. 5)

Lippman, S. A., A. M. Leddy, T. B. Neilands, J. Ahern, C. MacPhail, R. G. Wagner, D. Peacock, R. Twine, D. E. Goin, F. X. Gomez-Olive, A. Selin, S. M. Tollman, K. Kahn and A. Pettifor (2018). “Village community mobilization is associated with reduced HIV incidence in young South African women participating in the HPTN 068 study cohort.” Journal of the International Aids Society 21(e25182).

Community mobilisation. "The CM intervention, conducted in partnership with Sonke Gender Justice and carried out by a trained team of mobilizers and community volunteers, sought to address intersections around HIV risk and gender norms that contribute to gender-based violence and power inequities, encouraging community members to examine how to make changes in both their own lives and in their communities through workshops and varied community activities". (p. 61) "HPTN 068 (happening simultaneously) was a randomized trial of cash transfers conditional on school attendance." (p. 61)

See protocol for further details: Pettifor A, Lippman SA, Selin AM, Peacock D, Gottert A, Maman S, et al. A cluster randomized-controlled trial of a community mobilization intervention to change gender norms and reduce HIV risk in rural South Africa: study design and intervention. BMC Public Health. 2015;15:752.

“We examine the association of CM with incident HIV among AGYW (ages 13 to 21) enrolled in the HPTN 068 cohort in the Agincourt Health and socio-Demographic Surveillance System, South Africa.” (from abstract)

“Simultaneous to the HPTN 068 trial, a community mobilization programme and research initiative was underway at the Agincourt HDSS site, with implementation of a CM intervention in 11 of 22 randomly selected villages in the area. The CM intervention, conducted in partnership with Sonke Gender Justice and carried out by a trained team of mobilizers and community volunteers, sought to address intersections around HIV risk and gender norms that contribute to gender-based violence and power inequities, encouraging community members to examine how to make changes in both their own lives and in their communities through workshops and varied community activities. The intervention was evaluated using cross-sectional surveys conducted prior to (n = 1181) and following (n = 1403) the two-year intervention (2012 to 2014).” (p. 61) [process described further in protocol Pettifor et al 2015]

“We examine the association of CM with incident HIV among AGYW (ages 13 to 21) enrolled in the HPTN 068 cohort in the Agincourt Health and socio-Demographic Surveillance System, South Africa. This analysis includes 2292 participants residing in 26 villages where cross-sectional, population-based surveys were conducted to measure CM among 18- to 35-year-old residents in 2012 and 2014. HPTN 068 participants completed up to five annual visits that included an HIV test (2011 to 2016). Household-level data were collected from AGYW parents/guardians and census data is updated annually. Mean village-level CM scores were created using a validated community mobilization measure with seven components (social cohesion, social control, critical consciousness, shared concerns, organizations and networks, leadership and collective action). We used pooled generalized estimating equation regression with a Poisson distribution to estimate risk ratios (RR) for the association of village-level CM score and CM components with incident HIV infection, accounting for village-level clustering and adjusting for key covariates.” (from abstract)

Miller, R. L., S. J. Reed, D. Chiaramonte, T. Strzyzykowski, H. Spring, I. D. Acevedo-Polakovich, K. Chutuape, B. Cooper-Walker, C. B. Boyer and J. M. Ellen (2017). “Structural and Community Change Outcomes of the Connect-to-Protect Coalitions: Trials and Triumphs Securing Adolescent Access to HIV Prevention, Testing, and Medical Care.” American Journal of Community Psychology 60(1-2): 199–214

“The Connect-to-Protect (C2P) Partnership for Youth Prevention Intervention was a demonstration project of the Adolescent Medicine Trials Network for HIV/AIDS Interventions (ATN) and funded by the National Institute of Child Health and Development, National Institute on Mental Health, and National Institute on Drug Abuse.” (p.5) “Coalitions included representatives of private and public health-focused organizations, organizations from other youth-focused sectors (e.g., education, juvenile justice), and prominent community institutions (e.g., businesses, churches, mayor’s offices). Coalition partners included community-based organizations specializing in youth of color, GBLTQ youth, and HIV. In many cases, youth from the target population were also members of the coalitions. Coalitions approached mobilization and planning using a structured process adapted from Fawcett et al.’s (2000) VMOSA approach. The approach was modified to incorporate root cause analysis to guide the development of a logic model linking structural and community drivers of risk and structural risk mechanisms to individual youth behaviors. Coalitions used their logic models to develop structural change objectives corresponding to the drivers and mechanisms they identified. Each objective was further delineated into action steps with target completion dates and a list of other actors needed to move the objective forward. Structural changes targeted numerous sectors in the community, including healthcare, education, criminal justice, religious, and social services.” (p. 5–6)

“...to identify the features of coalitions’ context and operation that facilitated and undermined their ability to achieve structural change and build communities’ capability to manage their local adolescent HIV epidemic effectively. In the current study, we examine the perceived contributions and accomplishments of these coalitions at the end of their lifespans to identify the features of their context and operation that facilitated and undermined their ability to achieve structural change and build capability to effectively manage their local adolescent HIV epidemic.” (pp.4–5)

“Outcome mapping”: “To identify key informants who possessed specialized knowledge of either the effects of structural changes on the systems and sectors where these effects occurred or of the cascading effects of these changes on youth, coalition staff used outcome mapping techniques. We viewed outcome mapping as an appropriate tool because of its emphasis on capturing changes in the behavior, relationships, activities, or actions of the people, groups, and organizations with whom an entity such as a coalition works. In outcome mapping, these “boundary partners” are the people through which change occurs. It is their practices and the policies they must follow in carrying out their work that coalitions are seeking to influence via structural changes. Staff nominated 293 people in 2015 and an additional 168 people in 2016 as prospective informants, for a total of 461 potential interviewees.” (p. 6) “C2P staff running the coalitions and the staff at the NCC documented coalitions’ activities, member composition, member feedback, and the status of each structural change objective on an ongoing basis and in a standardized manner. During the final 2 years of the project (2014–2016), 318 key informant interviews with youth and community leaders were conducted by an external evaluation team based at Michigan State University” (p. 6) “Coalition leaders may have disproportionately recommended informants who possessed positively biased views of the coalition and a small number of informants who were likely to offer critical views on coalition accomplishments or functioning. […] We sought to guard against this possibility by asking for detailed descriptions of changes and actions, with a focus on observed changes in the 2 years prior to each interview. We asked for evidence in support of every claim of positive impact. We limited our analysis to changes that corresponded with accomplished objectives from the coalition’s and NCC’s records, as these could be most clearly attributed to the coalition’s work. Nonetheless, the sample of informants may have provided us with an unduly favorable view of the coalitions’ accomplishments, painting a picture of them as more successful and impactful than warranted.” (p. 18)

Nove, A., L. Hulton, A. Martin-Hilber and Z. Matthews (2014). “Establishing a baseline to measure change in political will and the use of data for decision-making in maternal and newborn health in six African countries.” Int J Gynaecol Obstet 127(1): 102-107.

E4A (see Hulton et al) “The Evidence for Action (E4A) program assumes that both resource allocation and quality of care can improve via a strategy that combines evidence and advocacy to stimulate accountability.” (from abstract)

“The questions for E4A therefore were: how could political will be measured; to what extent did decision-makers have access to and use data; and how could change over time in these two key outcomes be measured? To help answer them and determine the baseline situation for the program, we designed two tools: the Politics, Power, and Perceptions (PPP) tool and the Data for Decision-Making (DDM) tool” (p. 102)

Note: the authors report the ‘independent’ study was designed before the country teams had been recruited, limiting the ownership of the study by the country teams “and consequently much time and effort was required to explain the value of the data to them, and to encourage them to use the data to help plan their strategy” (p. 103)

Design: Repeat cross-sectional survey with repeat interviews with respondents in each phase where possible.

Sample: Purposive sampling “In each country, independent consultants were contracted to select and interview a purposive sample of 40–60 key informants for each tool, to gather views from an appropriate spread of national level, district level, and facility level informants. At national and district level (here district refers to the subnational level that was appropriate for each country), the pool of eligible informants was relatively small and the aim was to interview as many as possible. At facility level, the sampling was done by listing all possible health facilities in the E4A focal areas, then selecting a subsample based on how practical it was to visit them within the allotted time. At each sampled facility, contractors were instructed to interview 1–3 eligible informants according to the informants’ availability on the day of the visit.” (p. 103)

Data collection methods: two questionnaires. “The PPP tool assesses the level of political will to improve MNH outcomes. The DDM tool assesses the extent to which key stakeholders make use of MNH data. The use of face-to-face interviews allowed for a detailed set of questions (average interview duration was 20 min for PPP and 30 min for DDM) and for interviewers to request documentary evidence to back up the responses given by DDM informants, which acted as an important quality control mechanism. However, the use of a structured questionnaire meant informants’ answers could not be explored in more detail to gain more qualitative insight.” (p. 103)

Samuel, J. (2016). “The role of civil society in strengthening intercultural maternal health care in local health facilities: Puno, Peru.” Global Health Action 9(1)1.

“The initiative recruited, trained, and supported Quechua speaking indigenous women from community-based organizations (CBOs) in the department of Puno to act as volunteer citizen monitors to observe and report on the delivery of health care services in their local publicly provided facilities. Lawyers from the Puno office of the Defensor\( \overset{\overset{\acute{\mkern6mu}}{\kern0.333em }}{\mathrm{i}} \)a del Pueblo, Peru’s National Human Rights Ombudsman’s Office, also provided the monitors with training and support, as did other strategic allies.” (p. 2)

This article examines whether a grassroots accountability initiative based on citizen monitoring of local health facilities by indigenous women can help to promote the objectives of the intercultural birthing policy and improve intercultural maternal health care.

“The findings presented here are drawn from a larger qualitative research study that included fieldwork conducted in 2010 and 2011. Methodologically, this study used an institutional ethnographic approach to examine the work of citizen monitors in Puno, Peru. Institutional ethnography is based on the premise that analyzing the work processes and other experiences of a particular group of people can provide an important vantage point to understand a broader set of social and institutional relations. The author uses the notions of work processes and work knowledge to help explore and understand the work, roles, and working relationships of the citizen monitors in Puno. This involves an in-depth examination of the daily monitoring work done by this group of women to promote change in reproductive health service delivery. This approach is well suited to gain insight into the complex power relations that shape the monitors’ unequal engagement with their local health facilities.” (p. 3)

Schaaf, M., S. M. Topp and M. Ngulube (2017). “From favours to entitlements: community voice and action and health service quality in Zambia.” Health Policy Plan 32(6): 847–859.

“The primary objective of Citizen Voice and Action (CVA) is to increase dialogue and accountability between three groups: citizens, public service providers and government officials (political and administration) to improve the delivery of public services” (p.849) “The program occurs in three, iterative phases. The first phase entails World Vision (WV)-led relationship building with communities and service providers and stakeholder mobilization to inform the community and relevant actors about the goals and components of CVA. Next, WV convenes an open community gathering during which a CVA Committee is formed, usually by a consensus process. About 10–15 people join; membership is voluntary. CVA

Committee members are often also members of other community structures, such as village development committees and neighborhood health committees. Insofar as possible, WV tries to facilitate the creation of a diverse CVA Committee, so that the Committee has widespread legitimacy.

Following facilitation from WV, representatives from the government educate communities about relevant legislation and national service delivery standards. Citizens may have preferences and priorities that are not formally enshrined in national standards, thus they also articulate standards (“perception-based indicators”) that they think their local facility should meet. In the second phase, the health facility’s (or other service provider’s, depending on the context) realization of both perception-based indicators and national service delivery standards are assessed. A social audit process is used with service providers and communities to assess performance of the clinics against national service delivery standards. Here, citizens and service providers observe the facility and look at facility data to assess to what extent the facility is compliant with national service delivery standards. Then, citizens and service providers use community score cards to rate their health facilities against the perception-based indicators. Third, citizens, local elected representatives and service providers, convene interface meetings. They discuss the service delivery gaps identified and elaborate action plans to address some of these challenges. Action plans identify individuals and groups responsible for each action. The plans are then implemented and monitored in subsequent interface meetings. The three phases are repeated, as communities and the government tackle increasingly difficult challenges.” (p. 849)

“We sought to make tentative, contextualized programmatic and theoretical propositions about how the CVA program theory was realized in the health sector in 3 of Zambia’s 103 districts. The study aimed to answer:

1. How does CVA affect the relationship between citizens and the health sector?

2. How does the health sector respond to CVA?

3. What elements of context facilitate or hinder positive change in the health sector in response to CVA?” (p. 850)

“A full-fledged realist evaluation would typically require longitudinal engagement with program participants and stakeholders. Moreover, given CVA’s widespread use, a rigorous realist evaluation would entail looking at multiple countries. Thus, we describe this study as a realist informed qualitative study, an approach that has been taken in other contexts where researchers feel that the context, mechanisms, and outcomes framing would add value to extant data” (p. 850)

Data collection methods: “Secondary data were used iteratively. Secondary data included WV program documents, score cards and action plans generated by CVA activities, and materials WV developed summarizing health entitlements. More importantly, we also reviewed articles regarding social accountability in all domains (not just health), as well as health systems and policy research articles relating to relationships within health systems and between communities and the health system.” (p. 851).

“Primary data were collected between November 2013 and January 2014. CVA had started in these communities in 2008. At the time the research was conducted, the program was ongoing in all of them. Methods used included in-depth interviews with district health officials (n = 5), traditional community leaders (n = 2), rural health center staff from one facility in each of the three sites (n = 4), WV staff based in the districts under study (n¼8), and WV staff based in Lusaka (n = 1). Focus groups were also conducted with CVA members in each of the three sites (n = 27).” (p. 851)

Sebert Kuhlmann, A. K., S. Gullo, C. Galavotti, C. Grant, M. Cavatore and S. Posnock (2017). “Women's and Health Workers’ Voices in Open, Inclusive Communities and Effective Spaces (VOICES): Measuring Governance Outcomes in Reproductive and Maternal Health Programmes.” Development Policy Review 35(2): 289–311.

CARE Community Score Card (CSC): “The theory of change for the CSC is based on CARE International’s Governance Programming Framework (GPF). The adaptation of the GPF to maternal and reproductive health focuses on three key ingredients: 1) empowering women and service users, 2) empowering health care service providers, and 3) creating spaces for service users and service providers to engage in constructive dialogue and negotiation. “ (p. 291)

“A cluster-randomized trial of the CSC is being conducted in Malawi, focused on reproductive and maternal health service delivery and outcomes. As part of the baseline data collection for the randomized trial, measures of key components of the theory of change were developed. These measures are part of two multidimensional survey tools—one for women called Women’s Voices in Open, Inclusive Communities and Effective Spaces (VOICES) and another for health workers called Health Worker’s VOICES. These same survey tools, with some minor modifications resulting from the analyses presented here, will be used during the end line data collection for the randomized trial in order to assess change over time as a result of the CSC intervention.” (p. 292)

“We examined the psychometric properties of the measures in each of these domains using baseline data from the cluster-randomized trial in Malawi. Data presented here were collected in Ntcheu district, Central Region, Malawi from October to December 2012. The evaluation uses a cluster-randomized control design with ten matched pairs of health facilities and surrounding catchment areas. Matching criteria included presence (or absence) of basic emergency obstetric services, facility administrator (MOH or CHAM), proximity to the Mozambique border and population size of the catchment area. One of each pair was randomly assigned to the intervention arm. Two cross-sectional surveys, one of women and one of health workers were conducted at baseline. In the 20 catchment areas, Women’s VOICES surveyed women aged from 15 to 49 who had given birth within the past 12 months—regardless of whether they had delivered in a health facility or not—and whose babies were still living, using a two stage probability proportional to size (PPS) methodology […] All 327 health workers (both facility- and community-based) within the 20 catchment areas were eligible for the Health Workers’ VOICES survey.” (p. 295)

Topp, S. M., J. Black, M. Morrow, J. M. Chipukuma and W. Van Damme (2015). “The impact of human immunodeficiency virus (HIV) service scale-up on mechanisms of accountability in Zambian primary health centres: a case-based health systems analysis.” BMC Health Services Research 15: 67.

Paper describes scale up of HIV services, and looks at social accountability as part of that. The main mechanism for social accountability was Neighborhood Health Committees (NHCs)

“The explicit focus of this article is to examine whether and how the establishment and scale-up of HIV services influenced mechanisms of accountability within the primary service domain, and, as a result, service quality and responsiveness. We then apply these findings to a consideration of whether there is merit in attempting to design disease specific interventions that reflect the complexity in primary level services, and, in the process, enable a more contextually comprehensive approach to the design and implementation of health system strengthening interventions.” (p. 2)

“We adopted a multi-case study design using a theoretical replication strategy. Case ‘units’—four primary health centres located in two adjacent Districts, one urban one rural—were selected by the lead investigator (SMT) in consultation with District Medical Officers, and based on both empiric and anecdotal evidence of characteristics that enabled exploration of patterns of service delivery. Such characteristics included: average patient attendance data; vaccination coverage rates; and District officers’ descriptions of health center performance. […]Methods used at each case site included: in-depth interviews with a proportionate sample of healthcare workers from various levels (n = 60); semi-structured interviews with a quasi-random sample of patients (n = 180); review of health center paperbased registers; and direct observation of facility operations […] In addition, key informant interviews were held with government and non-government officials (n = 14) with specific knowledge or experience of the processes of HIV service scale-up.” (pp. 4–5)