To achieve meaningful and rapid reductions in tobacco related morbidity and mortality, we must increase both the number of individuals who attempt to quit, and the success rate of those attempts. Among the many available cessation programmes, telephone based cessation services, also known as quitlines, offer the opportunity to achieve both goals because of their broad reach and effectiveness.1^–5 Two Cochrane reviews have supported quitlines, particularly those that offer proactive services, as an effective population based approach to tobacco cessation.1 2 Studies have also shown quitlines to be cost effective6 and cost beneficial.7 In fact, quitlines are increasingly being recognised as a key component of comprehensive tobacco control strategies in Australia,8 Canada, the United Kingdom, the United States9 10 and the European Union.

The emergence of quitlines as a population based approach to smoking cessation has followed the classic innovation diffusion curve,11 at least in North America. The first North American quitline service at a population level began in California in 1992; by 2002 there were 33 state and provincial quitlines and by 2006 all smokers in North America had access to quitline services.12 Despite the proliferation of quitlines there were no agreed upon metrics or standards for evaluation, making it difficult for quitlines to learn from each other, and comparative studies impossible.

The first North American Conference of Smoking Cessation Quitlines was held in Arizona in 2002. Participants at that meeting asked for a common approach to evaluation. The North American Quitline Consortium (NAQC), which was established two years later to provide leadership and a unified voice for quitlines, led the initiative to create a common evaluation approach. This paper describes the development of an evaluation framework and standardised set of measures (now known as the minimal dataset or MDS) designed to inform quitline operations, facilitate comparative evaluation studies and enhance research opportunities.

EVALUATION FRAMEWORK

All evaluations are concerned with utility, generalisability, scientific rigor and relevance. While there are many approaches to evaluation, utilisation focused evaluation is distinguished by its emphasis on utility, relevance and stakeholder involvement.13 What fundamentally distinguishes utilisation focused evaluation is that “the evaluator does not carry the burden for making choices about the nature, purpose, content and methods of evaluation alone. These decisions are shared by an identifiable and organised group of intended users.” Guiding principles described by Patton include active involvement of users, a commitment to meeting multiple and varied needs, a concern for utilisation as a driving force, sensitivity to situational factors affecting utilisation and investment in working interactively with stakeholders.

This framework and accompanying principles were especially relevant for the creation of a common dataset for evaluating quitlines. There were a large and diverse number of stakeholders with an interest and enthusiasm for learning what works in what settings and for whom. The rich diversity of quitline service delivery models, clients and target populations in different settings across North America made comparative studies possible via the adoption of a set of common measures.

DEVELOPMENT PROCESS

The North American Quitline Consortium to establish a 14-member working group including quitline researchers, service providers, evaluators and funders from across North America. Members of the working group had linkages to leading edge quitlines and key organisational affiliations (National Cancer Institute, US Centers for Disease Control and Prevention, Health Canada, NCIC Centre for Behavioural Research and Program Evaluation). From the outset, the vision was to develop a set of measures that would provide valid, standardised data on a few important indicators.

The process, illustrated in table 1, occurred over two years. The working group operated via teleconference and email, with key face to face meetings attached to other meetings and conferences, all supported by funding from NAQC sponsors, Health Canada and the Canadian Tobacco Control Research Initiative. Members of the working group provided leadership and momentum for the MDS, both internationally and within their own organisations. In addition, researchers with the European Smoking Cessation Helplines Evaluation Research (ESCHER) group of the European Network of Quitlines (ENQ) participated in key meetings.

To begin, a generic logic model was created to clarify the causal relation between quitline inputs (for example, resources, staffing models), activities (for example, promotion, counselling protocols), reach (client characteristics) and outcomes (quit attempts, quit rates). The logic model helped identify areas for indicator development. Next, existing evaluation tools were gathered from Canadian and US quitlines and formed the pool from which the selection of indicators and items was made. This respected the previous work of individual quitlines. The third step was an extensive consultation process to get agreement on the evaluation domains, to differentiate between “essential” and “important, nice to have” indicators and to select relevant intake and follow-up questions. This process was critical to getting buy in from the diverse set of stakeholders interested in quitlines’ success. By linking domains, questions and decisions, quitline funders, service providers, evaluators and researchers could see the relevance and importance of each question and the benefits of standardising questions across quitlines.

The Working Group adopted a set of guiding principles to ensure the MDS would facilitate service provision, evaluation and research, make comparisons possible and not impose undue resource burdens on quitlines. These principles were operationalised as follows:

• indicators must inform decisions important to the improvement of quitlines

• whenever possible, preservation of questions and wording already in use to allow quitlines to continue historical comparisons

• preference given to measures with acceptable reliability and validity, endorsed by scientific bodies (for example, Society For Research On Nicotine And Tobacco, SRNT), or used in national surveys (for example, census demographic questions)

• size (total number of items) of the MDS must not create barriers to meeting the needs of smokers calling for help with quitting.

After decisions on evaluation domains and question wording were approved, the working group addressed protocols for data collection, issues pertaining to length of follow-up, repeat callers and information needed to understand the uniqueness of each quitline.

The final MDS and protocols were approved by the Research and Evaluation Working Group in February 2005. To facilitate implementation, NAQC shared the MDS with all NAQC members and interested parties by email, on the NAQC website and during the first NAQC annual meeting in May 2005. NAQC also hired a technical expert and offered assistance via conference calls, online resources and individual consultation for a six-month period.

By the end of 2005, all quitlines reported that voluntary implementation of the MDS was planned or under way. A NAQC survey of MDS implementation is planned for 2007 to determine implementation, fidelity, areas for update and new information needs.

MINIMAL DATASET

The minimal dataset consists of a set of 15 intake (baseline) questions asked of eligible callers at the time of their first call and eight follow-up questions collected during follow-up interviews seven months after intake. As per the logic model, indicators included those designed to evaluate quitline promotion activities (for example, client awareness and response to promotion strategies, previous calls to the quitline), reach (type of caller, client demographic characteristics, geographic location and tobacco behaviours), service delivery (client satisfaction) and outcomes (changes in tobacco behaviours, including quit attempts, quit rates and switching to other forms of tobacco). As with any evaluation, callers can refuse to answer questions without penalty, although almost all intake questions also informed the counselling intervention.

Three factors that have consistently been shown to predict quit success (nicotine dependence, readiness to quit, use of other quit aids) are also captured. Although many quitlines use stages of change to inform their counselling interventions, the working group did not include it in the MDS given the equivocal evidence of its’ predictive validity for quit success when other factors are taken into account.14^–16

Table 2 outlines the MDS intake and follow-up questions, the corresponding evaluation indicators and examples of the types of decisions informed by each domain. The MDS questions and technical documents are posted on the NAQC website at www.naquitline.org/index.asp?dbid = 2&dbsection = research.

In addition to the MDS questions, the working group recognised that accurate descriptions of quitlines would be required to understand evaluation results. Recommendations on key administrative and service data were added to the technical documents accompanying the MDS (posted at http://naquitline.org/index.asp?dbid = 3&dbsection = research).17 In 2005 NAQC revised its annual survey of North American quitlines and Cummins et al present results showing the tremendous variability in quitline models.18 These data reinforce the importance of context in evaluation. In future comparative studies, the MDS can be linked to (current) descriptive information about participating quitlines to help answer questions about the effect of different delivery models on abstinence.

There is also considerable variability in quitlines’ evaluation mandates and resources. With respect to evaluation methodology, the working group concluded: “Each quitline will need to determine how follow-up will be conducted and on which population. …[quitlines] should strive to survey enough people to draw valid conclusions about their outcomes, but it will be up to the individual quitline to determine whether census surveying, random sampling, cohort sampling, or some other sampling method will be most appropriate.”17 While not ideal from a research perspective, this approach made implementation of the MDS possible with the expectation that in future common evaluation methodologies could also be implemented.

The working group followed expert guidelines for abstinence measures,19 20 recommending at minimum, six-month follow-up and 30-day point prevalence. This measure was meaningful to funders and follow-up was feasible for quitlines. Quitlines already measuring prolonged abstinence and 12-month outcomes were encouraged to continue. NAQC also recommended that quitlines report abstinence rates using both an “intent to treat” analysis and analysis of only those who completed the evaluation, with the recognition that the true quit rate lies somewhere between the two measures. It was also determined that a one-month intervention period would permit completion of most or all proactive counselling calls for most quitlines. Thus the working group recommended the follow-up interviews occur seven months after the intake date. This allowed one month for the full intervention (or at least the majority of proactive calls), plus the standard six-month follow-up period.

The final issue tackled by the working group was how to treat repeat callers. A small percentage of callers make extensive use of quitlines and the dilemma was at what point to consider them as making a new quit attempt versus continuing to act on the original counselling intervention. After considerable debate, it was agreed that a 12-month period from the first quitline contact should be considered a new quit attempt. Thus smokers who relapsed but called back for a second quit attempt 13 or more months after their original call to the quitline would be considered as new callers for evaluation purposes.

DISCUSSION

The minimal dataset provides the basis for a North American laboratory or “community of practice” for research and evaluation and enables us to capitalise on the diversity across quitlines and the large numbers of callers served. Quitlines themselves represent a success story of the translation of research evidence to public health practice. The MDS represents a best practice in quitline evaluation. It is a living document and will be revised and expanded as evidence, experience and capacity allow.

A desire to keep the MDS small and operationally feasible took paramount importance in discussions. It was recognised that the MDS must be easy to implement and be respectful of quitlines’ service mandate. The multi-stakeholder working group enabled satisfactory resolution between researchers’ desire for comprehensive baseline data and service providers’ concerns about the time required to collect those data. As one participant noted “What made this work was willingness on the part of the researchers to balance scientific rigor with practicality, coupled with a respect for and interest in scientific integrity on the part of program and policy decision makers.”

A second key success factor was accommodating differences across quitlines. Individual quitlines each had their own evaluation questions to preserve and reasons for wanting to do so. Acknowledgement that the MDS was an adjustment to existing protocols authenticated individual quitline evaluation processes while at the same time moved the field as a whole toward a core set of items. The resulting MDS replaced some but not all pre-existing questions.

Though time intensive, the multi-stakeholder collaborative approach was another key success factor. As noted by one quitline service provider, “The collaborative approach was unique and progressive. By bringing together such a range of stakeholders, all perspectives were represented from the very beginning, which allowed informed and rapid input, critiquing and feedback to produce a MDS that will be relevant and manageable to implement”. Also important to success was the momentum, started by Ossip-Klein in a presentation at the North American Conference of Smoking Cessation Quitlines and continued by the co-chairs and members of the working group. Finally, NAQC’s role in communicating and providing technical assistance to address implementation issues and support quitlines was critical.

There are several early indicators of the success of this endeavour. As of December 2006, the majority of US states and almost all Canadian provinces are implementing the MDS at some level (Bailey, personal communication). Other indicators of success include adoption of the MDS as a template for data collection by the University of Massachusetts Tobacco Treatment Specialist Training Program. The US Centers for Disease Control and Prevention asked quitlines they fund to report on the use of the MDS.9 Other institutions have recognised the value of standardised data and are proposing to adapt the MDS or create a similar standardised dataset (Web Assisted Tobacco Intervention Initiative; University of Rochester Medical Center; the Ontario Tobacco Research Unit; and the Association for the Treatment of Tobacco Use and Dependence).

CONCLUSION

The North American Quitline Consortium developed, disseminated and oversaw implementation of a minimal dataset for quitlines. This was a joint effort of multiple stakeholders (service providers, funders, evaluators, researchers) in Canada and the United States, with advice from leading research groups in North America (SRNT) and Europe (ESCHER). This achievement should be useful to other areas of tobacco control programming, as the MDS represents a best practice in tobacco control evaluation. The NAQC encourages other components of comprehensive tobacco control strategies to consider building on this experience so we can accelerate reductions in tobacco related mortality and morbidity.