CENTER FOR THE ADVANCEMENT OF HEALTH

So let me, then, describe some of the different measures that have been proposed or are being used. I will begin with measures that were recommended by the Center for the Advancement of Health. The centre, with funding from the Robert Wood Johnson Foundation, put together an expert working group which came up with proposed measures that were submitted to the National Committee for Quality Assurance (NCQA) when it was considering the development of a HEDIS tobacco measure. I was one of the members of that working group. (NCQA is the organisation that accredits health maintenance organisations, and it has responsibility for overseeing HEDIS.)

One measure would provide for tobacco-use screening and intervention. The proposed measure was defined as the proportion of patients, ages 10 and older, who were screened and appropriately counselled in the reporting year. The method would involve a review of medical charts or computerised medical records. So this would be aprocess measure.

The screening portion of the measure would involve determining the person’s tobacco-use status (current, former, or never-user). The intervention portion of the measure would be based on the provider recording the type(s) of interventions used. Those interventions, in our proposed measure, would be the following.

• Was the patient advised to stop or avoid tobacco use at this visit? If the person was a tobacco user, was he or she advised to stop? If the person was a non-user, aged 10–21 years, then we felt that there needed to be some record of whether he or she received a message at that particular visit about not beginning tobacco use. So, was that person advised not to begin tobacco use at this visit?

• Was self-help material provided in the past 12 months?

• Was nicotine replacement therapy with a behavioural support programme prescribed in the past 12 months? This measure was proposed in the pre-Zyban era, so Zyban (bupropion) was not addressed.

• Was the patient referred to cessation follow up in the past 12 months? If so, please describe the follow-up action taken.

The working group also wanted to have anoutcome measure. The one we recommended—smoking prevalence—was one of the “test” measures proposed by HEDIS for further study (see below). The prevalence measure we suggested was the prevalence of current smokers among health plan members 10 years of age and older.

A third measure recommended by the working group was “medical advice to quit”, which, like the first suggested measure, is more process oriented. It was defined as the proportion of smokers who received medical advice to quit at least once during the past year—which is very similar to the reporting measure that HEDIS adopted. The working group proposed that both of these measures—smoking prevalence and medical advice to quit—would be calculated by means of a health plan survey, which is how the new HEDIS tobacco measure is assessed.

HEDIS

I presented background on the Health Plan Employer Data and Information Set (HEDIS) in an earlier supplement toTobacco Control, 1 so I’ll avoid doing so here.

However, I would like to draw your attention to the half-dozen preventive services covered in HEDIS. Cholesterol testing had been in HEDIS but was dropped, apparently because the rates were generally quite high and did not allow discrimination across managed care organisations. Pap tests, mammograms, and childhood immunisations were in HEDIS 2.5 (the earlier version) and remain in HEDIS 3.0 (the current version). Two preventive services added in HEDIS 3.0 are influenza immunisation for those 65 years of age and older and smoking cessation advice. Two other measures for preventive services have been added to HEDIS: varicella and hepatitis B immunisation of 13 year olds.

In HEDIS, advising smokers to quit is part of the “reporting” set. In other words, this is a measure that had to be reported in 1997—for having received advice to quit during 1996. HEDIS also proposed two measures for the “testing” set, which are being evaluated for inclusion in a future reporting set. Those test measures are the prevalence of current smokers (similar to the measure recommended by the Center for the Advancement of Health) and the “prevalence of quitting” during the past year.

I will focus on the “advising smokers to quit” measure because that is the one that is actually in operation now. The actual wording of the measure is:

“Among Medicaid, commercial, and Medicare risk enrolled adults, aged 18 and older as of December 31 of the reporting year, who were continuously enrolled during the reporting year, who were either current smokers or recent quitters, and who were seen by a plan provider during the reporting year—the percentage who received advice to quit smoking during the reporting year from a plan provider.”

Here is how the sampling and the methods are defined. The sampling would involve three separate samples, with a sample size of 1860 for each. The three samples are for commercial members, Medicaid members, and Medicare risk members (that is, Medicare beneficiaries in a capitated health plan). For the commercial members, the methods would involve asking questions (see below) in the annual member healthcare survey (often referred to as the annual member satisfaction survey), with the questionnaire being sent through the mail. The Medicare and Medicaid samples would be surveyed through the Consumer Assessments of Health Plans study of the Agency for Health Care Policy and Research.

The five questions included in the member satisfaction survey are as follows. The first two would define smoking status: (a) “Have you ever smoked at least 100 cigarettes in your entire life?”; and (b) (for those answering “yes” to the first question) “Do you now smoke every day, some days, or not at all?” If you do smoke everyday or some days, you are classified as a current smoker. These two questions therefore give you the prevalence of current smoking in your health plan. Corrine Husten made the point earlier in this conference that if a health plan is collecting information for the HEDIS tobacco measure, it can easily calculate smoking prevalence from these first two questions.

The third question (for former smokers) is: “How long has it been since you quit smoking cigarettes?” Those who have quit in the past year would be classified as a recent quitter and they would be included in the denominator of the final measure. In addition, that question would give you the data (numerator) to calculate the other test measure—the quit rate during the past year. Again, these two test measures—prevalence of current smokers and prevalence of quitting during the past year—can be calculated using the five questions that are already being used in the member satisfaction survey.

Question four is: “During the past 12 months, how many times have you visited a doctor or other health professional in your plan?” If you visited a health plan doctor in the past year at least once, then you are classified as having been seen in the plan during the past year; and so you remain in the denominator.

The final question is about advice to quit: “On how many of these visits were you advised to quit smoking by a doctor or other health professional in your plan?” If you received that advice to quit on one or more visits, then you are classified as having received medical advice to quit and then you jump up to the numerator. So that is how the reporting measure is calculated, through those five questions.

FACCT

Let me now review the relevant measures recommended by FACCT (the Foundation for Accountability). On a promotional flyer FACCT bills itself as a “consumer voice on healthcare quality”, which puts out measures “for those who buy and receive healthcare . . . who will take the lead in mapping a course for an accountable health system”. The foundation is governed almost entirely by purchasers and consumer representatives. It has very little health plan representation on its governing body, and it highlights that distinction between itself and the National Committee for Quality Assurance (NCQA), which the foundation believes is dominated or controlled by health plans. FACCT also believes that HEDIS is too process oriented, so FACCT strives to develop performance indicators that are more outcome oriented.

FACCT has recommended two tobacco measures: (a) provider advice/support to quit smoking (similar to the HEDIS measure), which they categorise in their “Steps to Good Care” domain; and (b) the smoking quit rate, which is in their “Results” domain. The “quit rate” measure is the type of outcome-oriented measure that they favour, and is similar to the measure in the HEDIS testing set. But FACCT, presumably, believes that this measure should be implemented now, without pilot-testing.

The methods they recommend involve a health risk survey akin to the behavioural risk factor survey supported by the US Centers for Disease Control and Prevention, which is conducted by telephone by all 50 states. FACCT recommends that the survey use the questions from the Center for the Advancement of Health’s expert working group.

The FACCT measures have been used far less than have the HEDIS measures. But they are important, in my opinion, because FACCT speaks in large part for purchasers, which, after all, are the ones who pay for health care and medical services.

SMALL SAMPLE SIZE

Although it is extremely valuable to have tobacco measures in HEDIS, we should acknowledge and address potential problems with those measures. First, there is a sample-size problem. As I mentioned earlier, the sample for this measure is 1860. Let us assume the following: (a) a 50% response rate to this mailed questionnaire; (b) that 20% of members are smokers or recent quitters (because one would expect smoking prevalence to be lower in managed care populations than in the general population, in which smoking prevalence is about 25%)5 ,6; and (c) that about 70% of smokers have seen a health plan provider in the past year.7 That would leave us with a denominator of 130, which is much less than what we would like to have (figure 3).

• Download figure
• Open in new tab
• Download powerpoint

Figure 3

Estimated sample sizes for the tobacco measure in the HEDIS reporting set, based on certain assumptions.

The figure of 20% for the prevalence of smokers (and recent quitters), as indicated in figure 3, refers to commercial members. For Medicaid members, the figure would be considerably higher. A survey of commercial and Medicaid members of eight health maintenance organisations in Michigan showed a smoking prevalence of 19.4% for the commercial members, 44.1% for the Medicaid members, and 25.9% for the general population in that state.5 So the sample size problem might be mitigated in the Medicaid population because of a higher smoking prevalence.

Here is what the NCQA says about sample size: “Sampling will be carried out to assure that at least 107 adult smokers who have seen a physician complete the questionnaire.”8 My assumptions gave me the figure of 130, so NCQA must have used slightly different assumptions to come up with 107. NCQA also says that “When the sample size is between 30 and 100, the measure will have little power to detect differences between plans that are smaller than 20 percentage points.”8 We are getting close to the point (sample size of 100 or less) where the statistical power for the HEDIS tobacco measure would be unacceptably low.

Figure 4 shows the experience at Health Alliance Plan (HAP), where I work. We actually had a 40% response rate. Seventeen per cent of respondents were smokers or recent quitters who had seen a health plan provider in the past year. So our final denominator (124) was slightly less than the one predicted (130) based on the assumptions shown in figure 3.

• Download figure
• Open in new tab
• Download powerpoint

Figure 4

Actual sample sizes for the tobacco measure in the HEDIS reporting set, for Health Alliance Plan (HAP) of Michigan for 1996.

HAP’s HEDIS rate was 70%—that is, a bit higher than the aggregate figure for the more than 300 plans submitting data to NCQA (61%). But note the wide 95% confidence interval around that rate (plus or minus eight percentage points). That wide confidence interval—62% to 78%—reflects the lack of precision for this measure.

RESPONSE BIAS

There is also a potential problem of response bias. Smokers may be less likely than non-smokers to respond to the survey. The potential for response bias is higher because the respondents are told that the health plan is the sponsor of the survey, and respondents may fear “de-selection” (cancellation of their coverage) or premium increases for unhealthy behaviours, even if they are assured at the outset of the questionnaire that this will not happen and that the results are anonymous.

RECALL BIAS

Another potential problem is the possibility of recall bias. An Australian study assessed the accuracy of patient recall of physician advice to quit smoking, compared with a “gold standard” of audiotape analysis of 1075 patient consultations involving 43 postgraduate general practitioner trainees. Smokers over-reported advice to stop smoking.9 Patient recall of having received advice to quit had a specificity of 82%, a sensitivity of 92%, a positive predictive value of 42%, and a negative predictive value of 99%. Positive predictive value is the percentage of positive results that are actually positive, and negative predictive value is the percentage of negative results that are actually negative. In other words, a negative report of advice to quit is quite accurate, but a positive report is no better than 50–50 in accuracy.9 ,10That would certainly raise questions about the validity of this HEDIS measure. The measure may still be useful for assessing performance within a given health plan over time if the degree of that recall bias does not change, but there may be a problem of comparability across health plans because of that potential bias.

TOBACCO MEASURES IN THE TESTING SET

There are a few problems with the tobacco measures recommended for the testing set. For the smoking prevalence measure, one problem is that interventions are less likely to measurably affect prevalence than other indices of smoking such as cigarette consumption. An example is the “Fairness Doctrine” public service announcements. In the late 1960s, television and radio stations in the United States were required to broadcast anti-smoking spots under the Federal Communications Commission’s “Fairness Doctrine” as long as cigarette commercials were still on the air. These anti-smoking messages resulted in one of the largest drops in cigarette consumption ever reported, but there was little or no detectable decrease in smoking prevalence during that period.11

Sometimes prevalence will not be a sensitive indicator for measuring the impact of interventions. Intermediate effects, such as reducing daily cigarette consumption or advancing smokers through the early stages of readiness to quit,12 will not be detected in a prevalence measure.

In addition, prevalence reflects many societal factors, such as demographics, legislation, and religion. Utah, for example, has the lowest smoking prevalence among all 50 states,13 mainly due to its large Mormon population. Health plans have little or no direct control over these societal factors.

The NCQA has commented on this problem:

“The impact that plans can make on prevalence may be as low as 1% per year. Thus, the use of smoking prevalence to distinguish between plans needs to be assessed. A risk-adjustment strategy may be needed to enable this measure to be used for plan comparisons. These issues, among others, will be evaluated during the testing phase.”8

The test measure for prevalence of quitting has a few problems similar to those for the smoking prevalence measure. Intermediate effects of interventions, such as reducing smokers’ daily cigarette consumption or moving people through the stages of readiness to quit, will not be detected. In addition, smoking cessation rates reflect many societal and environmental factors over which health plans may have little direct control. Again, NCQA has acknowledged these limitations:

“Plans may experience success at first, with smokers who are less entrenched in the habit. However, over time, a plan’s success may diminish as it tries to influence the more hard-core smokers. Because plans will be at different stages in their efforts, a risk-adjustment strategy may be needed to make this a valid measure for comparing between plans. These issues will be evaluated during the testing phase.”8

These problems or potential problems with the HEDIS measures—including the measure in the reporting set and the two measures in the testing set—highlight the need for more research and evaluation concerning the measures and, if appropriate, refinement of the measure now being used in the reporting set.