Article Text
PDF
Abstract
Aim We conducted this study to determine key community-level factors associated with higher tobacco control programme performance.
Methods A combination of surveys, administrative and fiscal data were collected to measure local county-level health department performance over a 7-year period. Longitudinal analyses were performed using generalised estimating equations to examine whether counties that exerted higher effort were successful in creating more tobacco retail licensing (TRL) and secondhand smoke policies. Several social, political and contextual factors were examined as confounders.
Results Local county health departments (CHDs) that demonstrated high effort on their work plans increased the proportion of residents covered by TRL policies (7.2%; 95% CI −1.7 to 16.1%) compared to CHDs with lower levels of effort. Having legislators who voted in favour of tobacco control bills was found to significantly increase the passage of local TRL policies. CHDs demonstrating higher efforts also increased the proportion of residents covered by secondhand smoke policies (9.2%; 95% CI −3.5 to 21.9%).
Conclusion There was strong evidence that higher county-level efforts predicted an increasing number of local tobacco control policies. Evaluations using integrated designs are recommended as effective strategies to provide a more accurate assessment of how well community-level interventions catalyse community-wide change.
- Community program intervention studies
- health promotion
- outcome assessment
- smoking/prevention and control
- California
- environment
- public policy
- surveillance and monitoring
This is an open-access article distributed under the terms of the Creative Commons Attribution Non-commercial License, which permits use, distribution, and reproduction in any medium, provided the original work is properly cited, the use is non commercial and is otherwise in compliance with the license. See: http://creativecommons.org/licenses/by-nc/2.0/ and http://creativecommons.org/licenses/by-nc/2.0/legalcode.
Statistics from Altmetric.com
- Community program intervention studies
- health promotion
- outcome assessment
- smoking/prevention and control
- California
- environment
- public policy
- surveillance and monitoring
Introduction
The California Department of Public Health's California Tobacco Control Program (CTCP) encompasses a diversity of efforts from the coordination of 61 local county health departments (CHDs), to over 40 community-based organisations, a statewide media campaign, a tobacco cessation helpline and provision of technical assistance services. Community-level interventions are the building blocks of the CTCP. According to the CTCP's ‘social norm change’ approach, durable public health related social norm change is achieved at the population level through shifts in the social environment of local communities.1 The CTCP realises this by funding programmes and projects that achieve lasting change with lower cost community-level preventive tobacco control outcomes, rather than through costly and ineffectual individual-level remedial programmes or policies. California's community-level tobacco control interventions that are funded by the state primarily focus on four policy themes: (1) mobilising the community to create and implement local ordinances that restrict locations where smoking can occur; (2) countering the tobacco industry's marketing practices; (3) reducing access to tobacco products, and; (4) increasing access to cessation services.
Previously, it has been demonstrated that the California programme has successfully decreased smoking prevalence,2–5 morbidity6–8 and tobacco-related healthcare expenditures.9 A shift from individual tobacco control interventions towards population-based interventions came with increased funding for comprehensive tobacco control programmes during the late 1980s.10 11 The emergence of statewide coalitions has been cited as the most important advance in comprehensive state programmes, such as those in California and Massachusetts.12 From its inception, California has benefited from working with grassroots efforts at the local level to adopt an increasing number of local ordinances and creating hundreds of tobacco control policies (Francis JA, et al. See pages 16–20 of this supplement). Without the passage of over a hundred clean air ordinances in cities across the state, California's Smokefree Indoor Workplace Law would have been unlikely to have been enacted and enforced.1 Even so, the evidence for the impact of the community component of comprehensive tobacco control programmes is less robust.
Public health experts concur that the link between community action and changes in programme outcomes is difficult to measure because public health methodology is not as well developed for measuring complex community-based and population-based social policy change as it is for individual and small group change.11 13 14 There have been two state-sponsored independent evaluations of the CTCP.15–18 The first, a process evaluation, occurred during the programme's earliest years (1990–1994) to assess programme inputs, such as staffing, types of activities, and characteristics of programme participants.16 This evaluation found a shift from the individual to community-level coalition and advocacy work but had no means of quantitatively demonstrating the contribution of programme efforts to declines in smoking prevalence. The second, an outcome-focused evaluation, also occurred during the programme's early years (1996–1998) to determine relationships between programme inputs, exposure and programme outcomes using a repeated cross-sectional design at the county-level of analysis.5 It found that the CTCP's early efforts may have reduced ultimate programme outcomes, such as smoking prevalence and exposure to secondhand smoke. The final report of this outcome evaluation study included an additional 2 years of data (1996–2001) and drew similar conclusions.19 Several limitations of these outcome evaluations have been noted, such as relying on individual-level recall of programme components, not adjusting for any confounders when testing multiple non-independent hypotheses in regression models, and examining only 18 of the 58 counties. The Centers for Disease Control and Prevention, through evidence-based analyses, in-depth involvement with settlement States and published evidence of effective tobacco control strategies, strongly recommends funding community-level interventions in its Best Practices for Comprehensive Tobacco Control Programs.20 Yet, generally there is no strong quantitative evidence supporting the effectiveness of community-wide interventions in tobacco control. This remains a large gap in the evaluation of the CTCP.
State-funded community programmes in California's 61 local CHDs are charged with implementing their own local comprehensive tobacco control plan, to coordinate and ensure the success of all tobacco control efforts undertaken in their jurisdictions.1 Having the commitment of key community representatives, in the role of community leaders, law makers, enforcement agencies, city officials and voluntary health organisations is an expression of social will and has become an increasingly important focus of comprehensive tobacco control efforts.11 21 22 As tobacco industry contributions increase, a legislator's tobacco policy tends to be more pro-tobacco—demonstrated by campaign contributions being significantly related to tobacco policy scores even after accounting for constituent attitudes and personal characteristics of legislators (Garrett JD, Francis JA, Weich Reusche K. 2010. (Submitted)).23 This suggests that industry contributions are a measure of tobacco industry involvement in policy interventions independent of the ‘social will’ of constituents.21 22 As a result of the CTCP taking a strong anti-industry stance at the outset of the programme, new coalitions were formed, partnerships emerged, and the anti-tobacco tobacco movement transcended the programme.1
Although the CTCP's last independent evaluation attempted to quantify the link between community-level inputs and programme outcomes, we attempt to overcome its limitations with the present study. We explore the impact of California's funded community programmes on intermediate outcomes. Our logic model (figure 1) illustrates our underlying framework. In California's 58 counties, we investigate the relationship between community efforts and intermediary programme outcomes from 2001 to 2007. Importantly, we assess pro-tobacco industry efforts as confounders. We hypothesise that counties that demonstrate higher work plan implementation and effort will show: (1) an increased proportion of residents exposed to strong tobacco retail licensing (TRL) policies and; (2) an increased proportion of residents exposed to strong secondhand smoke (SHS) policies in outdoor public places.
Logic model framework for California tobacco control policy outcomes.
Methods
This paper was restricted to 58 of the 61 local CHDs that oversee California's 58 counties because three local health departments are city rather than county-based. Local health departments in California are funded by the CTCP to carry out tobacco control activities.
Outcome variables
We obtained data on TRL policies in California, 2001–2008, from California's Center for Tobacco Policy and Organizing (CTPO). These policies were adopted at the city-level and county-level. Strong TRL policies were defined as having the following components:
Requirement that all retailers that sell tobacco products obtain a license and renew it annually.
Sufficiently high fee to fund effective administration and enforcement efforts, including compliance checks.
Coordination of tobacco regulations such that any violation of existing local, state or federal tobacco regulation violates the license.
Financial deterrent through fines and penalties including suspension and revocation of the license.
We obtained data on secondhand smoke policies for outdoor public places in California, 1998–2008, from the Americans for Nonsmokers' Rights Foundation's US Tobacco Control Laws Database.24 Some examples of areas covered by these policies include outdoor dining and shopping areas, service lines, parks and beaches. We excluded policies exclusively covering enclosed public spaces such as retail stores, bowling centres, transit areas, multi-unit housing and day-care centres.
TRL and SHS city-level policy data were imported into SAS V.9.1 and assigned to counties. Using US census data, we created a variable to calculate the proportion of a county's population covered by a TRL or SHS policy each year. We then calculated the cumulative proportion of a county covered by TRL and SHS policies during four time periods that reflect CHD work plan periods: (1) before 1 July 2001; (2) between 1 July 2001 and 30 June 2004; (3) between 1 July 2004 and 30 June 2007; (4) after 1 July 2007.
Independent variables
Implementation quality
We surveyed CTCP's programme consultants who oversee CHDs to collect data on the implementation quality of a county's work plan. Tobacco control programme consultants answered four questions (Likert-scale responses) related to each 3-year work plan to assess county-level implementation:
The completed intervention activities have been conducted effectively.
During the plan period, appropriate changes had been made to improve the interventions if they were needed.
The progress reports were well prepared.
Appropriate deliverables were submitted with progress reports.
We slightly modified the language of the four items to improve reliability of the scale for the second work plan (Cronbach's α: 0.74 vs 0.91 (for 1st and 2nd plan periods)). Since tobacco control interventions require a considerable amount of implementation to be successful, CTCP consultants held each work plan to a similarly high standard when they responded to the survey.
We created an indicator variable from the above four questions to compare CHDs without appropriate work plan implementation to those that appeared to have better implementation. The z scores of the combined four questions were used to determine cut-off levels for stronger county-level implementation.
Budget deliverable effort
To support and integrate local efforts and to put theory into practice, the CTCP created an Online Tobacco Information System (OTIS). This system provides staff the ability to provide intensive technical assistance and real-time monitoring of activities and spending. The CTCP requires each local CHD to submit a 3-year work plan that focuses on the most locally relevant tobacco control issues. Work plans include primary objectives and deliverables budgeted to each objective. These plans are then peer-reviewed and negotiated with CTCP. Beginning in 2001, all local CHD contracts, including work plans and budget, were entered into OTIS.
We used two funding cycles for this study (2001–2004 and 2004–2007), during which each local CHD received a designated allocation of funds from CTCP. The per cent budgeted deliverable for each objective related to the two policy outcomes (TRL or SHS) were extracted from OTIS and imported into SAS for analysis.
Confounders
As described earlier and illustrated in our logic model, pro-tobacco industry efforts are seen as an important influential factor on tobacco control efforts. To determine whether counties were more successful at passing policies as a result of influential covariates, we examined the influence of social, political, and contextual variables on community-wide tobacco control efforts (figure 1). These variables included CHD capability, county-level demographics and pro-tobacco efforts.
Social and political variables were compiled from a number of sources. To assess CHD capability we used the number of full time employees and turnover in lead tobacco control staff (yes/no) from OTIS work plans. We obtained the following census-tract level aggregate variables from the California Department of Finance, Demographic Research Unit25: race/ethnicity (ie, non-Hispanic Whites, Hispanics, Asians), gender-age (ie, <45 years old, ≥45 years old), education (%<high school, %≥high school, %some college, %≥college), employment/income (%unemployed, %household income below poverty, per capita income), metropolitan residency (%urban, %rural) and total population of each county. We used county-level variables as numerators and total county population as the denominator to create county-level proportions.
Pro-tobacco industry efforts were measured using three variables. We used legislator voting data and monies contributed to legislators, 2001–2007, from CTPO. Assembly votes were calculated separately from senate votes since geographic areas represented by senate districts are different from those of assembly districts. Senate and assembly district may cross county lines or be totally encompassed within a county. Consequently, we created a county level variable for each year that was a weighted average of the assembly person's or senator's voting records based on the proportion of the population represented by each legislator. The final variable was the sum of the proportion of assembly and senate votes in favour of tobacco-related legislation weighted to the population in a given county. We weighted the amount of money given to legislators similarly.
The CTCP regularly observes a random sample of sporting events that are thought to have tobacco sponsorship. These include auto races, motorcycle races, bull riding events, rodeos, and team roping events. Actual numbers of tobacco industry sponsored events by county were incorporated into the final analysis.
Data analysis
We strengthened a ‘theory of change’ approach26 by integrating it with quasi-experimental methodology.27 28 Without this, it would have been difficult to interpret our findings in the context of community-level interventions. This ‘integrated design’ is similar to that described by evaluators in other fields.28 29
Predictors and influential factors
We collapsed the two variables that assessed intervention effort, measured by per cent budgeted deliverable effort and implementation quality of an intervention. We simplified the analysis by dichotomising the continuous variable for budgeted deliverable effort to compare those with no effort to those with some effort dedicated to each indicator. The final ‘effort’ variables may be visualised using a two-by-two table with implementation quality and budgeted deliverable effort forming the row and column headings. The four cells of this table include: (1) strong implementation and any budgeted effort to an intervention; (2) weak implementation with any budgeted effort; (3) strong implementation but no budgeted effort; (4) weak implementation and no budgeted effort. After examining impact on programme outcomes, we found that having a weak implementation or no effort resulted in poorer outcomes that were statistically indistinguishable, regardless of the outcome measure. Consequently for the final model, we decided to collapse the last three levels of the strength of tobacco control measure because the last three levels were not meaningfully different. This created a dichotomous measure of county-level effort for analysis: (1) ‘strong counties’ with strong implementation and any budgeted effort versus; (2) ‘weak counties’ with weak implementation or no budgeted effort.
Differences in means of the comparison counties by influential factors (ie, sociodemographic and political variables) were examined using a t test. Categorical variables were examined using a χ2 test or Fisher's exact test, when appropriate.
Generalised estimating models models
Generalised estimating equations (GEE)30 for continuous outcomes with the identity link was used for analysis. We determined whether there were any changes in dependent variables before entering them into GEE models. The exchangeable correlation structure adjusted the standard errors for multiple observations per county. The QIC (quasi-likelihood under the independence model criterion) statistic was used to confirm that the exchangeable correlation structure assumption was appropriate.
Separate models were computed for TRL and SHS policy outcomes. The base model contained only work plan implementation and budgeted deliverable effort. We examined sociodemographic, pro-tobacco industry, and local CHD variables as potential confounders. We used separate models for the two policy outcomes to determine the presence of potential sociodemographic and political confounding variables. Due to a large number of potential confounders, we entered each into the base model using stepwise forward selection (using α=0.05 for inclusion and α=0.1 for exclusion) followed by a simultaneous elimination method to determine factors associated with each of the two programme outcomes.
First, candidate variables (p<0.1) were selected as potential confounders in a stepwise fashion. At each step, variables were dropped from the model based on coefficient p-values. The above steps were repeated in an iterative process to create the final model. The GENMOD procedure was used to calculate GEE parameter estimates.
Statistical analyses were performed using SAS V.9.1. The criterion of statistical significance was set at α=0.05 (two-sided).
Results
Changes in outcome variables
Counties with stronger work plan implementation and higher budgeted deliverable effort were termed ‘strong counties’. Between 2001 and 2007, a significantly greater proportion of strong counties (33.2%) were covered by TRL policies as compared to remaining counties (table 1). By 2007, strong counties had an increase in the cumulative percentage of the population covered by outdoor SHS policies in public areas (25.5%). This was significantly greater than that observed in remaining counties (6.8%).
Mean changes in cumulative percentage of population covered by TRL and SHS policies from 2001 to 2007
Changes in independent variables
The quality of work plans significantly increased (8.5; 95% CI: 8.2, 8.9) compared to baseline (8.1; 95% CI: 7.7, 8.4). Also the average budget that was dedicated to each of the objectives increased: TRL policies increased (4.2% at baseline versus 12.8% at follow-up) and SHS policies targeting public areas increased (5.8% versus 10.3%). Using the combined strength of tobacco control variable for budgeted deliverables and implementation quality, we found that 17 local CHDs exerted better effort during the second work plan period compared to the first work plan period. This resulted in 32 of the 58 local CHDs exhibiting strong effort by the second plan period.
Changes in confounders
We found declines in pro-tobacco industry efforts over time (table 2). The average proportion of bills in the state legislature that favoured strong tobacco control legislation increased (0.5 vs 0.7), the amount of tobacco industry money contributed to legislators significantly decreased (US$40.91 vs US$30.27 per 1000 persons), and the number of tobacco industry sponsored sporting events also significantly decreased over time (3.2 vs 1.9 events). This decline in pro-tobacco monies to legislators and increase in the proportion of bills favouring stronger tobacco control legislation may be explained by the decline in spending after two major spikes during 1993–1994 and 1997–1998. These spikes in lobbying expenditures occurred during periods when significant state legislation on smoke-free laws had been proposed. Yet tobacco industry efforts were only successful at delaying policies, which may explain the resulting decline in spending.
Mean changes in pro-tobacco industry efforts from 2001 to 2007
We did not observe any changes in county-level socio-demographics or local CHDs staffing variables over time.
Did strong work plan implementation and effort predict increased TRL policy population coverage?
Before adjusting for any confounders, simply having a strong work plan implementation and budgeted deliverable effort dedicated to TRL was strongly associated with having a greater percentage of the population covered by TRL policies (table 3). None of the measures of tobacco industry involvement were significant predictors of TRL population coverage when the strength of tobacco control measure was entered as a four-level categorical variable. After adjusting for voting history, counties that had high quality implementation and higher effort addressing TRL increased the percentage of their population covered by a strong TRL policy on average by 7.2% (95% CI: −1.7 to 16.1%) compared to other counties.
Impact of county-level efforts on increasing tobacco retail licensing (TRL) policies
Did strong work plan implementation and effort predict increased SHS policy population coverage?
In the base model, work plan implementation and effort were not significant predictors of increasing outdoor public SHS policies (table 4). Two measures of tobacco industry involvement, voting history and monies given to legislators, were no longer significant predictors when entered together into the model with the four-level effort variable. None of the other potential confounders were significant predictors of SHS policies. After we dichotomised the measure of programme effort, no confounders remained in the model. In the final model, we found counties that had strong work plan implementation and higher effort on outdoor public SHS objectives tending to increase population coverage of outdoor public SHS policies by 9.2% (95% CI: −3.5, 21.9) compared to remaining local CHDs.
Impact of county-level efforts on increasing outdoor public secondhand smoke (SHS) policies
Discussion
Our evaluation suggests a relationship between county effort and policy outcomes, accounting for geographic variation, using quantitative measures and analyses. Over a 7-year period, we found that local CHDs were more successful in creating policies if they engaged in stronger implementation efforts. There was evidence that an increased effort towards TRL policies resulted in a greater proportion of the population being covered by these policies. However, background political factors were found to significantly impact these local TRL efforts. A likely interpretation for this finding is that local CHDs who raised public awareness and influenced legislators were more likely to show an increase in TRL policies over time. It is likely that the impact of community efforts on TRL policies would have been even greater if legislators had voted in favour of tobacco control policies. Although there were no significant increases in SHS policies, local efforts were in the right direction.
This evaluation would not have been possible during the first decade of the programme. Since CTCP chose to use an online administrative system (ie, OTIS) to collect administrative data in 2001, this more effectively monitored local efforts and spending. In turn, real-time tracking offered the opportunity for more objective evaluation of community efforts.
In adopting the ‘theory of change’ approach, we decided which outcomes to examine based on what became salient during the last decade of CTCP planning. We used a comparison group, without which we would have likely concluded that CTCP's goals had not been met. The comparison group allowed efforts in similar counties to be more apparent, as well as appropriately identify the impact of community efforts. If we had decided to rely solely on the more traditional difference-in-differences analysis, evaluation outcomes would have been chosen during the earlier years of the programme. The integrated approach to evaluation, relying on both the theory of change and quasi-experimental approaches implicitly recognises the inevitable changes to programme and context.29
There are several opportunities for refining similar evaluations of community-level interventions. Our analysis grouped counties that were similar in their efforts. Still it may be a concern that counties with the greatest effort had been conferred an advantage from the beginning, by being ready for normative change. In particular, larger counties such as Los Angeles and San Francisco demonstrated high quality implementation throughout the observation period. By matching on observable factors, we introduced imprecision by ignoring unobservable differences between the two groups. However, this level of evaluation is still viewed as more sophisticated as compared to evaluation methodologies that only seek to monitor programmes or compare the performance of ‘assisted’ versus ‘typical’ ventures.31 32
Future evaluations may benefit from better proxy measures of pro-tobacco industry efforts. In our final model we used state legislator votes as a proxy measure of how the tobacco industry exerts influence at the county level. We considered using statewide ballot initiatives (Propositions 99, 10, 86) similar to Klink's Social Will Index,21 but felt that these initiatives reflected a willingness to raise taxes rather than a reflection of comprehensive tobacco control efforts. It seemed reasonable that state legislator votes on state tobacco control policies reflected key opinion leader support or opposition to county tobacco control efforts. State legislators predominately begin as local elected officials serving on county-level Boards of Supervisors, City Councils, or School Boards. Second, state legislators vote on bills that mirror the types of policies and initiatives funded by CTCP at the local CHDs. Since tobacco control efforts do not occur in a vacuum, it follows that tobacco industry influence at the county-level will counter or dilute comprehensive tobacco control efforts. Since we did not have other quantifiable proxy political indicators available at the county-level, we used state legislator votes on tobacco control related bills. Future evaluations may consider using behavioural or social indicators such as tobacco consumption data, percentage of smokers in a community, and number of hours devoted to legislator education.
Our evaluation was limited by the nature of the data. Several measures were imprecise because we relied on self-reports of local effort. We used multiple data sources for community-level outcomes as well as inefficient measures of important confounders. Policy outcomes used in our evaluation were provided by two separate community organisations rather than obtained through a single source. TRL policies were better defined than SHS policies, which likely impacted the validity of our estimates. This introduces the problem of information bias due to incomplete monitoring and collection efforts.
We augmented administrative data on local efforts by surveying programme consultants at the CTCP who oversee each local CHD. These questions relied on self-reports of a consultant's perceptions of conditions and performance. We attempted to correct for response bias by combining survey data with more objective budgeted deliverable efforts. This combined measure was the primary data source for predictors in our model. We hope to improve our programme consultant survey in the future by using externally validated questions and by assessing local efforts aimed at specific objectives rather than a local CHD's overall work plan. Unfortunately, self-reports are not easily avoided. Since comprehensive community interventions29 rely on community engagement, evaluating these efforts requires collecting perceptions of local conditions, norms and attitudes.
Finally, community-level evaluation required splitting resources between collecting and analysing survey and administrative/fiscal data. This was true even with California's large administrative system for tracking local efforts. Administrative data collection at the state level remains a costly and complex endeavour, but is a requirement for integrative design.
Despite a lack of evidence quantifying the impact of tobacco control community-level interventions, comprehensive community interventions have become very popular. Our analysis implies that it is possible for decision makers to quantify community-level impact using reliable and robust measures of effort. With continued declines in tobacco control programmatic spending, it has become increasingly important for funders and managers to justify supporting certain programme objectives over others. Sophisticated evaluations can provide a more accurate assessment of how well interventions catalyse specific community-wide changes. While we have attempted to narrow the gap between recommended tobacco control policy interventions and the lack of quantifiable evidence for these initiatives, much remains to be done.
What this paper adds
Over time, the CTCP's local CHDs were encouraged to take a more data-driven, evidence-based approach in creating local policies. We examine the impact of local CHD efforts on two tobacco-related policy outcomes over a 7-year period. Our findings suggest that, strongly implementing tobacco control work plans with high effort will result in greater tobacco retail licensing (TRL) and SHS policies in outdoor public areas. The political environment was shown to significantly impact the increase in TRL policies. More sophisticated evaluations of community-wide tobacco control interventions are recommended to strengthen the evidence for these local efforts.
Acknowledgments
We are grateful to Juan Yang for the countless hours she spent compiling datasets during the first plan period.
References
Footnotes
Competing interests None.
Provenance and peer review Not commissioned; externally peer reviewed.