Integrating a Comparison Group Design into a Theory of Change Evaluation: The Case of the Urban Health Initiative

doi:10.1016/S1098-2140(02)00236-9

The American Journal of Evaluation

Volume 23, Issue 4, Winter 2002, Pages 371-385

https://doi.org/10.1016/S1098-2140(02)00236-9 Get rights and content

Abstract

This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. We also demonstrate the plausibility of selecting a credible comparison group through the use of cluster analysis, and describe our work in validating that analysis with additional measures. The integrated evaluation design relies on two points of comparison: (1) program theory to program experience; and (2) program cities to comparison cities. We describe how we are using this integrated design to evaluate the Robert Wood Johnson Foundation’s Urban Health Initiative, an effort that aims to improve health and safety outcomes for children and youth in five distressed urban areas through a process of citywide, multi-sector planning and changed public and private systems. We also discuss how the use of two research frameworks and multiple methods can enrich our ability to test underlying assumptions and evaluate overall program effects. Using this integrated approach has provided evidence that the earliest phases of this initiative are unfolding as the theory would predict, and that the comparison cities are not undergoing a similar experience to those in UHI. Despite many remaining limitations, this integrated evaluation can provide greater confidence in assessing whether future changes in health and safety outcomes may have resulted from the Urban Health Initiative (UHI).

Section snippets

INTRODUCTION

This paper describes how we strengthened the theory of change approach to evaluating a complex social initiative by integrating it with a quasi-experimental, comparison group design. The Urban Health Initiative (UHI), sponsored by the Robert Wood Johnson Foundation (RWJF), aims to improve the health and safety of children and youth in distressed urban areas. Several features of the initiative, including its non-prescriptive guidelines, its focus on entire cities, and its emphasis on changing

THE URBAN HEALTH INITIATIVE

In developing and launching their UHI in 1995, the RWJF recognized that although a number of promising approaches to improving the health and safety of children had emerged, most had succeeded either in improving outcomes in a single, categorical area of concern or in improving the health and safety climate of a single neighborhood. None had been sufficiently broad-based to significantly improve the health and safety of young people, measured against a number of indicators, across an entire

THE EVALUATION OF UHI

Researchers at New York University were enlisted to evaluate this national initiative from the onset. The purposes of the national evaluation were to determine whether and how the UHI sites were able to effect change on a range of health and safety outcomes for young people over the course of the initiative. The evaluation design is guided by three research questions:

•
To what extent, and in what ways, can a foundation-sponsored initiative serve as a catalyst for a cross-sector collaborative

USING CLUSTER ANALYSIS TO IDENTIFY A COMPARISON GROUP

In order to blend theory-driven and comparison group approaches, we believed it essential that some of the baseline features of the cities participating in UHI be integrated into the selection of the comparison cities. The small sample of UHI cities (5) necessitated the use of a matched comparison group; if the UHI and comparison cities had been chosen at random from the top 100 cities, there is a good chance that the program cities and comparison cities would differ considerably. For example,

CHECKING THE VALIDITY OF THE COMPARISON GROUP

In response to the previously described criticisms made by Hollister and Hill (1995), we chose to check the validity of our selection in regard to a number of additional program-relevant measures. We used documents from, and our observations of, the RWJF selection process to provide some insight into their ideas of the “target group”—that is, the cities to which the findings of this evaluation might be generalizable. RWJF chose cities with demonstrable levels of distress among families and

USING AN INTEGRATED APPROACH: PRELIMINARY FINDINGS

As already noted, evaluators need to consider the counterfactual, not only at baseline and with regard to the outcomes of an intervention, but also throughout the progress of the intervention itself. Efforts like the UHI have, as their premise, a vision of altering complicated social, political, and economic arrangements. Such arrangements are fluid and dynamic, and evaluators seeking to understand interventions aimed at altering these arrangements face very difficult problems of attribution.

DISCUSSION AND CONCLUSION

Evaluation research is at the mercy of the counterfactual. Even in a theory of change evaluation, the need to ask whether the documented changes would have occurred in the absence of the program begs for comparison. At the same time, with initiatives as complex as the UHI, it is imperative to understand the expectation of program planners and implementers and to test their assumptions as part of the evaluation design. The two approaches must be integrated, so that evaluators can build the case

Acknowledgements

This work is funded by a grant from the Robert Wood Johnson Foundation.

References (22)

The Brookings Institution. (1998). Learning what works: Evaluating complex social interventions. Washington, DC: The...
Chen, H. (1990). Theory-driven evaluations. London: Sage...
Cook, T. D. (2000). The false choice between theory-based evaluation and experimentation. Program theory evaluation:...
Granger, R. C. (1998). Establishing causality in evaluations of Comprehensive Community Initiatives. In K....
Gruenewald, P. J., Treno, A. J., Taff, G., & Klitzner, M. (1997). Measuring community indicators: A systems approach to...
Hacsi, T. A. (2000). Using program theory to replicate successful programs. Program theory evaluation: Challenges and...
Hatry, H., Winnie, R. E., & Fisk, D. (1981). Program analysis for state and local governments (2nd ed.). Washington,...
Hauser, R. M., Brown, B. V., & Prosser, W. R. (Eds.). (1997). Indicators of children’s well-being. New York: Russell...
Hollister, R. G., & Hill, J. (1995). Problems in the evaluation of community-wide initiatives. In J. Connell, A....
Kubisch, A. C., Fullbright-Anderson, K., & Connell, J. P. (1998). Evaluating Community Initiatives: A progress report....

Kubisch, A., Weiss, C., Schorr, L., & Connell, J. (1995). Introduction. In J. Connell, A. Kubisch, L. Schorr, & C....

Cited by (41)

Community pandemic prevention and control measures and their influence on citizen satisfaction during the COVID-19 pandemic in China
2023, International Journal of Disaster Risk Reduction
Citation Excerpt :
Inadequate resources will make it difficult to provide comprehensive measures [79]. While its problem is not the oversupply of services but the basic provision of measures in rural [80]. The local government should strictly control the dissemination of information and develop an official information platform that integrates the government and the community.
At the forefront of the fight against the pandemic, the community’ s measures and services would have a greater impact than ever before on citizen satisfaction. However, the influence of citizen satisfaction on community pandemic prevention and control measures (CPPCM) during the pandemic is poorly understood. This study aims to investigate the allocation of CPPCM and its impact on CS. The Chinese national data was analyzed for the outcome. (1) Pandemic prevention propaganda (PPP), disinfection (DT), and body temperature tests (BTTs) were the primary measures taken by the Chinese community. (2) The CS for pandemic prevention and control is high, and urban and central Chinese communities express greater satisfaction. (3) The impact of disinfection, body temperature tests, free supplies, and assistance purchasing supplies on CS was greater in rural areas than in urban areas. (4) Regional variations exist in the impact of CS on CPPCM. (5) The number of measures has an inverted U-shaped relationship with citizen satisfaction. This study also suggests that the government should disseminate information about pandemic prevention in a timely manner, provide basic health and medical services, and evaluate the measures taken to avoid the discount effect.
Deliberations of the expert advisory council on innovation fund applications
2018, Zeitschrift fur Evidenz, Fortbildung und Qualitat im Gesundheitswesen
Der Innovationsfonds schafft Anreize zur Entwicklung und Erprobung von Versorgungsinnovationen im Bereich der gesetzlichen Krankenversicherung (GKV) mit dem Ziel einer qualitativen Weiterentwicklung der Versorgung in Deutschland. Über einen Zeitraum von zunächst vier Jahren (2016-19) werden jährlich 300 Millionen Euro für Projekte zu „Neuen Versorgungsformen“ und zur „Versorgungsforschung“ bereitgestellt. Der vom Bundesministerium für Gesundheit (BMG) berufene zehn-köpfige Expertenbeirat begutachtet in einem formalisierten Verfahren alle Anträge anhand verschiedener Kriterien auf wissenschaftliche Güte, Innovationsgehalt, Versorgungsrelevanz und Implementierbarkeit. Das vorliegende Diskussionspapier legt wichtige Überlegungen zur Antragstellung und Begutachtung dar und stellt diese zur Diskussion.
(Wie von den Autoren eingereicht)
The “Innovation Fund” provides incentives for the development and testing of healthcare innovations in the area of the statutory health insurance with the aim to improve the quality of care in Germany. Over a period of initially four years (2016-19), 300 million Euro will be allocated annually to projects on “innovative forms of healthcare provision” and “health services research”. Using a formalized procedure, the ten-member expert advisory board appointed by the German Federal Ministry of Health (BMG) assess all applications on the basis of various criteria for scientific quality, potential of innovation, relevance for health service delivery, and implementability. The present discussion paper sets out important considerations for submission and assessment and puts them up for discussion.
(As supplied by the authors)
Program theory evaluation: Logic analysis
2011, Evaluation and Program Planning
Program theory evaluation, which has grown in use over the past 10 years, assesses whether a program is designed in such a way that it can achieve its intended outcomes. This article describes a particular type of program theory evaluation—logic analysis—that allows us to test the plausibility of a program's theory using scientific knowledge. Logic analysis is useful for improving the intervention or finding alternatives for achieving intended outcomes; it influences the choice of type of evaluation to conduct and strengthens the validity of subsequent evaluations.
The objective of this article is to present the methodological principles and the roots of this type of program theory evaluation. We illustrate two types of logic analysis with two actual evaluation cases. There are very few published examples of program theory evaluation. This article will provide evaluators with both theoretical and practical information to help them in conceptualizing their evaluations.
Theory of Change from formulation to evaluation: A critique on the operationalization of the logic model
2023, Strategic Thinking, Design and the Theory of Change: A Framework for Designing Impactful and Transformational Social Interventions
The Development and Use of a Theory of Change to Align Programs and Evaluation in a Complex, National Initiative
2019, American Journal of Evaluation
Using evaluability assessment to assess local community development health programmes: a Scottish case-study
2017, BMC Medical Research Methodology

View all citing articles on Scopus

View full text