Statistics from Altmetric.com
In this panel session I will be making an introductory presentation, after which we will hear from Tim McAfee and Jodi Jessen. After all of the presenters have been heard from, we will address “questions for the future”.
In my talk I would like to review: (a) the value of performance measurement and the different ways in which performance measurement can serve our purposes; (b) the different tobacco measures that have been developed or proposed, including those proposed by the Center for the Advancement of Health, those being used in the Health Plan Employer Data and Information Set (HEDIS), and those proposed by the Foundation for Accountability (FACCT); (c) results of the HEDIS tobacco measure in the 1996 reporting set; and (d) problems or potential problems with the HEDIS measures.
The value of performance measurement
Performance measures can help us monitor change over time for an individual health plan. They can help us observe differences across health plans and disparities within health plans—for example, between commercial members and Medicaid or Medicare members. Performance measurement allows us to identify providers with the best and worst performance—at the level of the clinic and the level of individual providers. And finally it permits us to assess the effectiveness of interventions.
Once we have decided that we want to measure performance, the question is, how do we do it? As the cartoon in figure 1 shows, we have to determine whether we are making any impact. We have to decide which measures to use. Which measures should we use for these howling wolves? Should it be how high the moon rises in the evening sky, or how brightly the moon shines, or how much the stars are flickering? Those are the kinds of questions that we struggle to answer when we try to address tobacco in managed care.
CENTER FOR THE ADVANCEMENT OF HEALTH
So let me, then, describe some of the different measures that have been proposed or are being used. I will begin with measures that were recommended by the Center for the Advancement of Health. The centre, with funding from the Robert Wood Johnson Foundation, put together an expert working group which came up with proposed measures that were submitted to the National Committee for Quality Assurance (NCQA) when it was considering the development of a HEDIS tobacco measure. I was one of the members of that working group. (NCQA is the organisation that accredits health maintenance organisations, and it has responsibility for overseeing HEDIS.)
One measure would provide for tobacco-use screening and intervention. The proposed measure was defined as the proportion of patients, ages 10 and older, who were screened and appropriately counselled in the reporting year. The method would involve a review of medical charts or computerised medical records. So this would be aprocess measure.
The screening portion of the measure would involve determining the person’s tobacco-use status (current, former, or never-user). The intervention portion of the measure would be based on the provider recording the type(s) of interventions used. Those interventions, in our proposed measure, would be the following.
Was the patient advised to stop or avoid tobacco use at this visit? If the person was a tobacco user, was he or she advised to stop? If the person was a non-user, aged 10–21 years, then we felt that there needed to be some record of whether he or she received a message at that particular visit about not beginning tobacco use. So, was that person advised not to begin tobacco use at this visit?
Was self-help material provided in the past 12 months?
Was nicotine replacement therapy with a behavioural support programme prescribed in the past 12 months? This measure was proposed in the pre-Zyban era, so Zyban (bupropion) was not addressed.
Was the patient referred to cessation follow up in the past 12 months? If so, please describe the follow-up action taken.
The working group also wanted to have anoutcome measure. The one we recommended—smoking prevalence—was one of the “test” measures proposed by HEDIS for further study (see below). The prevalence measure we suggested was the prevalence of current smokers among health plan members 10 years of age and older.
A third measure recommended by the working group was “medical advice to quit”, which, like the first suggested measure, is more process oriented. It was defined as the proportion of smokers who received medical advice to quit at least once during the past year—which is very similar to the reporting measure that HEDIS adopted. The working group proposed that both of these measures—smoking prevalence and medical advice to quit—would be calculated by means of a health plan survey, which is how the new HEDIS tobacco measure is assessed.
I presented background on the Health Plan Employer Data and Information Set (HEDIS) in an earlier supplement toTobacco Control, 1 so I’ll avoid doing so here.
However, I would like to draw your attention to the half-dozen preventive services covered in HEDIS. Cholesterol testing had been in HEDIS but was dropped, apparently because the rates were generally quite high and did not allow discrimination across managed care organisations. Pap tests, mammograms, and childhood immunisations were in HEDIS 2.5 (the earlier version) and remain in HEDIS 3.0 (the current version). Two preventive services added in HEDIS 3.0 are influenza immunisation for those 65 years of age and older and smoking cessation advice. Two other measures for preventive services have been added to HEDIS: varicella and hepatitis B immunisation of 13 year olds.
In HEDIS, advising smokers to quit is part of the “reporting” set. In other words, this is a measure that had to be reported in 1997—for having received advice to quit during 1996. HEDIS also proposed two measures for the “testing” set, which are being evaluated for inclusion in a future reporting set. Those test measures are the prevalence of current smokers (similar to the measure recommended by the Center for the Advancement of Health) and the “prevalence of quitting” during the past year.
I will focus on the “advising smokers to quit” measure because that is the one that is actually in operation now. The actual wording of the measure is:
“Among Medicaid, commercial, and Medicare risk enrolled adults, aged 18 and older as of December 31 of the reporting year, who were continuously enrolled during the reporting year, who were either current smokers or recent quitters, and who were seen by a plan provider during the reporting year—the percentage who received advice to quit smoking during the reporting year from a plan provider.”
Here is how the sampling and the methods are defined. The sampling would involve three separate samples, with a sample size of 1860 for each. The three samples are for commercial members, Medicaid members, and Medicare risk members (that is, Medicare beneficiaries in a capitated health plan). For the commercial members, the methods would involve asking questions (see below) in the annual member healthcare survey (often referred to as the annual member satisfaction survey), with the questionnaire being sent through the mail. The Medicare and Medicaid samples would be surveyed through the Consumer Assessments of Health Plans study of the Agency for Health Care Policy and Research.
The five questions included in the member satisfaction survey are as follows. The first two would define smoking status: (a) “Have you ever smoked at least 100 cigarettes in your entire life?”; and (b) (for those answering “yes” to the first question) “Do you now smoke every day, some days, or not at all?” If you do smoke everyday or some days, you are classified as a current smoker. These two questions therefore give you the prevalence of current smoking in your health plan. Corrine Husten made the point earlier in this conference that if a health plan is collecting information for the HEDIS tobacco measure, it can easily calculate smoking prevalence from these first two questions.
The third question (for former smokers) is: “How long has it been since you quit smoking cigarettes?” Those who have quit in the past year would be classified as a recent quitter and they would be included in the denominator of the final measure. In addition, that question would give you the data (numerator) to calculate the other test measure—the quit rate during the past year. Again, these two test measures—prevalence of current smokers and prevalence of quitting during the past year—can be calculated using the five questions that are already being used in the member satisfaction survey.
Question four is: “During the past 12 months, how many times have you visited a doctor or other health professional in your plan?” If you visited a health plan doctor in the past year at least once, then you are classified as having been seen in the plan during the past year; and so you remain in the denominator.
The final question is about advice to quit: “On how many of these visits were you advised to quit smoking by a doctor or other health professional in your plan?” If you received that advice to quit on one or more visits, then you are classified as having received medical advice to quit and then you jump up to the numerator. So that is how the reporting measure is calculated, through those five questions.
Let me now review the relevant measures recommended by FACCT (the Foundation for Accountability). On a promotional flyer FACCT bills itself as a “consumer voice on healthcare quality”, which puts out measures “for those who buy and receive healthcare . . . who will take the lead in mapping a course for an accountable health system”. The foundation is governed almost entirely by purchasers and consumer representatives. It has very little health plan representation on its governing body, and it highlights that distinction between itself and the National Committee for Quality Assurance (NCQA), which the foundation believes is dominated or controlled by health plans. FACCT also believes that HEDIS is too process oriented, so FACCT strives to develop performance indicators that are more outcome oriented.
FACCT has recommended two tobacco measures: (a) provider advice/support to quit smoking (similar to the HEDIS measure), which they categorise in their “Steps to Good Care” domain; and (b) the smoking quit rate, which is in their “Results” domain. The “quit rate” measure is the type of outcome-oriented measure that they favour, and is similar to the measure in the HEDIS testing set. But FACCT, presumably, believes that this measure should be implemented now, without pilot-testing.
The methods they recommend involve a health risk survey akin to the behavioural risk factor survey supported by the US Centers for Disease Control and Prevention, which is conducted by telephone by all 50 states. FACCT recommends that the survey use the questions from the Center for the Advancement of Health’s expert working group.
The FACCT measures have been used far less than have the HEDIS measures. But they are important, in my opinion, because FACCT speaks in large part for purchasers, which, after all, are the ones who pay for health care and medical services.
Results of the HEDIS tobacco measure (1996)
NCQA has consolidated HEDIS data from more than 300 health plans (covering more than 37 million Americans) for the year 1996, and has published aggregate data for several HEDIS measures. Figure 2 provides data for the tobacco measure, from an NCQA document on the world wide web.2
The aggregate result is 61% for smokers or recent quitters having received advice to quit from a health plan provider in the past year. NCQA shows a wide range of performance, from 30% to 85%, and regional differences (ranging from 67% for New England to 56% for the Mountain region).
NCQA provides a comparison figure of 37% for the fee-for-service sector, citing an article from the Morbidity and Mortality Weekly Report 3 as the source of that figure. The figure of 37% is from a national population-based survey (National Health Interview Survey—Health Promotion and Disease Prevention supplement), which means that it is not, strictly speaking, just for the fee-for-service sector. However, because only 20% to 25% of the general population at the time of that survey (1991) were in managed care, the figure is probably reasonably close to the actual figure for the fee-for-service population.
NCQA has now reported aggregate data for 1997 from 447 managed health care plans covering more than 60 million Americans. Their more recent report indicates that the tobacco measure increased slightly from 61% for 1996 to 64.0% for 1997.4 The report noted that “If industry-wide performance were brought up to the 90th percentile benchmark of 74.3%, an additional 4.2 million enrollees who smoke would be advised about the benefits of quitting, and nearly 26,000 more people would quit smoking each year, saving hundreds of lives and saving tens of millions of dollars in health care costs.”4
Problems with the HEDIS measures
SMALL SAMPLE SIZE
Although it is extremely valuable to have tobacco measures in HEDIS, we should acknowledge and address potential problems with those measures. First, there is a sample-size problem. As I mentioned earlier, the sample for this measure is 1860. Let us assume the following: (a) a 50% response rate to this mailed questionnaire; (b) that 20% of members are smokers or recent quitters (because one would expect smoking prevalence to be lower in managed care populations than in the general population, in which smoking prevalence is about 25%)5 6; and (c) that about 70% of smokers have seen a health plan provider in the past year.7 That would leave us with a denominator of 130, which is much less than what we would like to have (figure 3).
The figure of 20% for the prevalence of smokers (and recent quitters), as indicated in figure 3, refers to commercial members. For Medicaid members, the figure would be considerably higher. A survey of commercial and Medicaid members of eight health maintenance organisations in Michigan showed a smoking prevalence of 19.4% for the commercial members, 44.1% for the Medicaid members, and 25.9% for the general population in that state.5 So the sample size problem might be mitigated in the Medicaid population because of a higher smoking prevalence.
Here is what the NCQA says about sample size: “Sampling will be carried out to assure that at least 107 adult smokers who have seen a physician complete the questionnaire.”8 My assumptions gave me the figure of 130, so NCQA must have used slightly different assumptions to come up with 107. NCQA also says that “When the sample size is between 30 and 100, the measure will have little power to detect differences between plans that are smaller than 20 percentage points.”8 We are getting close to the point (sample size of 100 or less) where the statistical power for the HEDIS tobacco measure would be unacceptably low.
Figure 4 shows the experience at Health Alliance Plan (HAP), where I work. We actually had a 40% response rate. Seventeen per cent of respondents were smokers or recent quitters who had seen a health plan provider in the past year. So our final denominator (124) was slightly less than the one predicted (130) based on the assumptions shown in figure 3.
HAP’s HEDIS rate was 70%—that is, a bit higher than the aggregate figure for the more than 300 plans submitting data to NCQA (61%). But note the wide 95% confidence interval around that rate (plus or minus eight percentage points). That wide confidence interval—62% to 78%—reflects the lack of precision for this measure.
There is also a potential problem of response bias. Smokers may be less likely than non-smokers to respond to the survey. The potential for response bias is higher because the respondents are told that the health plan is the sponsor of the survey, and respondents may fear “de-selection” (cancellation of their coverage) or premium increases for unhealthy behaviours, even if they are assured at the outset of the questionnaire that this will not happen and that the results are anonymous.
Another potential problem is the possibility of recall bias. An Australian study assessed the accuracy of patient recall of physician advice to quit smoking, compared with a “gold standard” of audiotape analysis of 1075 patient consultations involving 43 postgraduate general practitioner trainees. Smokers over-reported advice to stop smoking.9 Patient recall of having received advice to quit had a specificity of 82%, a sensitivity of 92%, a positive predictive value of 42%, and a negative predictive value of 99%. Positive predictive value is the percentage of positive results that are actually positive, and negative predictive value is the percentage of negative results that are actually negative. In other words, a negative report of advice to quit is quite accurate, but a positive report is no better than 50–50 in accuracy.9 10That would certainly raise questions about the validity of this HEDIS measure. The measure may still be useful for assessing performance within a given health plan over time if the degree of that recall bias does not change, but there may be a problem of comparability across health plans because of that potential bias.
TOBACCO MEASURES IN THE TESTING SET
There are a few problems with the tobacco measures recommended for the testing set. For the smoking prevalence measure, one problem is that interventions are less likely to measurably affect prevalence than other indices of smoking such as cigarette consumption. An example is the “Fairness Doctrine” public service announcements. In the late 1960s, television and radio stations in the United States were required to broadcast anti-smoking spots under the Federal Communications Commission’s “Fairness Doctrine” as long as cigarette commercials were still on the air. These anti-smoking messages resulted in one of the largest drops in cigarette consumption ever reported, but there was little or no detectable decrease in smoking prevalence during that period.11
Sometimes prevalence will not be a sensitive indicator for measuring the impact of interventions. Intermediate effects, such as reducing daily cigarette consumption or advancing smokers through the early stages of readiness to quit,12 will not be detected in a prevalence measure.
In addition, prevalence reflects many societal factors, such as demographics, legislation, and religion. Utah, for example, has the lowest smoking prevalence among all 50 states,13 mainly due to its large Mormon population. Health plans have little or no direct control over these societal factors.
The NCQA has commented on this problem:
“The impact that plans can make on prevalence may be as low as 1% per year. Thus, the use of smoking prevalence to distinguish between plans needs to be assessed. A risk-adjustment strategy may be needed to enable this measure to be used for plan comparisons. These issues, among others, will be evaluated during the testing phase.”8
The test measure for prevalence of quitting has a few problems similar to those for the smoking prevalence measure. Intermediate effects of interventions, such as reducing smokers’ daily cigarette consumption or moving people through the stages of readiness to quit, will not be detected. In addition, smoking cessation rates reflect many societal and environmental factors over which health plans may have little direct control. Again, NCQA has acknowledged these limitations:
“Plans may experience success at first, with smokers who are less entrenched in the habit. However, over time, a plan’s success may diminish as it tries to influence the more hard-core smokers. Because plans will be at different stages in their efforts, a risk-adjustment strategy may be needed to make this a valid measure for comparing between plans. These issues will be evaluated during the testing phase.”8
These problems or potential problems with the HEDIS measures—including the measure in the reporting set and the two measures in the testing set—highlight the need for more research and evaluation concerning the measures and, if appropriate, refinement of the measure now being used in the reporting set.
Let me conclude by showing you a page out of Health Alliance Plan’s HEDIS report for 1996 (which came out in mid-1997).14 You can see, at the bottom of the page, the results for our smoking measure, which I shared with you earlier. Our tobacco measure (70%) is now shown prominently in our HEDIS report, which we provide to purchasers, members of the public, those in the media, and anyone else who is interested. An introductory paragraph before the 70% figure speaks to the importance of addressing smoking and the impact it has on public health. This was not in our earlier HEDIS reports, when there was no tobacco measure in HEDIS.
The new reference to smoking in HAP’s HEDIS report demonstrates an important benefit of the tobacco measure in HEDIS. Despite its limitations, the measure is motivating some health plans to begin to address the problem and to highlight its importance to purchasers, members of the public, and other interested parties. Research on the measures, and refinement of them, will allow us to gain even greater benefit from them.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.