Article Text

Download PDFPDF

Is the standard compliance check protocol a valid measure of the accessibility of tobacco to underage smokers?
  1. Joseph R DiFranza,
  2. Judith A Savageau,
  3. Joseph Bouchard
  1. Department of Family Medicine and Community Health, University of Massachusetts Medical School, Worcester, Massachusetts, USA
  1. Joseph R DiFranza MD, Department of Family Medicine and Community Health, University of Massachusetts Medical School, 55 Lake Avenue North, Worcester, MA 01655, USAdifranzj{at}


OBJECTIVE To determine if the standard compliance check protocol is a valid measure of the experience of underage smokers when purchasing tobacco in unfamiliar communities.

SETTING 160 tobacco outlets in eight Massachusetts communities where underage tobacco sales laws are vigorously enforced.

PROCEDURE Completed purchase rates were compared between underage smokers who behaved normally and inexperienced non-smoking youths who were not allowed to lie or present proof of age (ID).

RESULTS The “smoker protocol” increased the likelihood of a sale nearly sixfold over that for the non-smokers (odds ratio (OR) 5.7, 95% confidence interval (CI) 1.5 to 22). When the youths presented an ID with an underage birth date, the odds of a completed sale increased dramatically (OR 27, 95% CI 3.4 to 212). Clerks judged to be under 21 years of age were seven times more likely to make an illegal sale (OR 7.6, 95% CI 2.4 to 24.0).

CONCLUSIONS Commonly used compliance check protocols are too artificial to reflect accurately the experience of underage smokers. The validity of compliance checks might be improved by having youths present ID, and by employing either tobacco users, or non-tobacco users who are sufficiently experienced to mimic the self confidence exhibited by tobacco users in this situation. Consideration should be given to prohibiting the sale of tobacco by individuals under 21 years of age.

  • compliance check protocol
  • underage smokers

Statistics from

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

In 1987, the compliance check was introduced as a research tool for measuring the availability of tobacco to minors.1 In the compliance check, an underage youth attempts to purchase tobacco from a commercial outlet. The compliance check has been used extensively as an evaluation tool in studies concerning both tobacco and alcohol, and is federally mandated as the official method by which state compliance with the Synar Amendment is measured.2-9 Compliance checks are now performed in every US state and territory, Canada, Britain, and Australia.10-13

The measured compliance rate can be influenced by a variety of factors (see Forster and Wolfson14 for a thorough review). Violation rates increase with the age of the youth.15-18Most studies report slightly higher sales rates for girls.15 16 19 20 In an uncontrolled study of 150 000 checks, girls were able to purchase in 26.3% of attempts versus 25.2% for boys.15 Boys are more often asked for proof of age.19 The effects of race and ethnicity are mixed.20-22 White and Hispanic adolescents were less likely to report being asked for proof of age than blacks in one study, but in another no such racial or ethnic effects were seen.21 22 A sale appears more likely when the clerk and buyer are of different races.23 Experience may be an important factor: three youths who completed 100 compliance checks each over a two month period showed a significant increase in their rate of successful purchases over time.24 A completed sale is more likely if the tobacco is obtained by self service rather than from a clerk.25 26 Forster found that sales were more common when one or more customers were waiting in line behind the youth.26

An additional factor with potential influence on the outcome of compliance checks is the youth's behaviour. Compliance check protocols commonly place constraints on the behaviour of the underage shopper.10 26 27 Typical protocols prohibit the participation of youths who smoke or who appear older than average; the use of measures such as makeup or jewellery to present a mature appearance; the misrepresentation of age; presentation of true or false identification (ID); or the use of any story, plea or conversation intended to persuade the clerk. Some protocols prohibit the youth from completing the purchase.28 These constraints may render the compliance check too artificial to reflect accurately the experience of underage smokers. The purpose of this study is to determine if a typical compliance check protocol accurately reflects the experience of underage smokers who are behaving naturally.



Paired compliance tests were conducted in a carefully controlled design to compare the rate of completed sales for inexperienced non-smoking youths following a standard protocol and for young smokers doing what they usually do to buy tobacco in a strange community.


A sample size of 160 tobacco outlets was selected to ensure that a 10% difference in measured compliance rates could be detected with an α error of < 5%, and a β error of < 20%.

A convenience sample of eight Massachusetts communities was selected from among those with a minimum of 20 vendors, and merchant compliance rates at or above 90% as measured by the local board of health during enforcement inspections. State and local laws prohibit the sale of tobacco to anyone under 18 years of age. The purpose of this study was not to compare compliance rates between communities or between types of outlets, but rather, between types of buyers. The communities were all suburbs and had a combined total of 351 tobacco vendors (range 21–83). Once the communities were selected, the particular outlets were chosen based upon their proximity to major roads because of the need to visit each outlet in one community twice within the same day. We had no knowledge of the outlets' past compliance. An average of 20 vendors were sampled in each community (range 17–24). Vending machines were excluded because buyer characteristics would be expected to be unimportant in the case of an unsupervised vending machine. The outlet types were distributed as follows: convenience stores 32%, gas stations 25%, liquor stores 15%, pharmacies 12%, gas mini-mart 6%, supermarket 5%, delicatessens 4%, and news stands 1%.


Two attempts to purchase tobacco were made from each of 160 outlets. One attempt entailed a standard enforcement-type protocol employing an underage non-smoker. The other attempt to purchase from the same outlet was made by a smoker of the same age, sex, and race/ethnicity using his or her customary purchasing strategies. Both attempts were made on the same day to maximise the similarity between the two attempts concerning the day of the week, the time of day, and the clerk. It was not possible to ensure that the same clerk serviced both youths. The two attempts were separated by at least 30 minutes to avoid suspicion. Order effects were avoided by having the smoker and non-smoker alternate in making the first purchase attempt at each outlet.

Violation rates may be related to whether other customers are waiting in line, therefore, we ensured that there was always another customer standing in line behind the youth by having one of three adult supervisors (one male and two females) enter the store separately from the youth and get in line behind the youth. The two pretended to be strangers. Individual supervisors alternated entering with the smokers and non-smokers to avoid introducing bias. Since many Massachusetts communities restrict self service displays, this portion of the protocol was kept constant between the two conditions by having all youths request cigarettes from the clerk.

The “standard,” or “non-smoker,” protocol employed non-smoking youths. Only one of the five non-smokers had prior experience with purchasing tobacco for a health department. The youths were instructed not to wear heavy makeup or to take other measures to appear older. The youths stated their correct age if asked. If asked for ID, they replied that they had none. They asked for a pack of cigarettes from the clerk, and if refused, left quietly.

The “smoker” protocol allowed youths to appear as they chose. All youths reported current tobacco use but data were not collected about smoking rates or past experiences with purchasing tobacco. They were allowed to purchase other items along with the tobacco if that was their custom. They were free to lie about their age or to present their own ID if they had one. They were not limited in what they could say to the clerk except that they were not allowed to make any threats, express anger, or use profanity. They were not allowed to use a false ID. All dressed as they normally would without any special attempt to appear older. Only one girl had an ID.

In both protocols, the youths asked for any brand they wished. The youths completed the purchases when permitted. This study was approved by the committee for the protection of human subjects in research at the University of Massachusetts Medical School. Parental permission was obtained for all youths employed in the study. The identities of the stores have been kept confidential to protect their rights as subjects.

Two boys and eight girls, ages 15–17 years, were employed. Although not by design, all of the youths were non-Hispanic whites, and as such they were not out of place in any of the eight communities which were all predominantly white. Four youths inspected only one community, six inspected two communities. The youths were instructed as to the protocol each was to follow but there were no practise purchases or role playing. None of the youths knew the store clerks. Both attempts to purchase at each outlet were made by youths of the same age, sex, ethnicity, and race. One pair of youths was randomly assigned to complete all of the checks in each community. Data recorded by the adult supervisor included the following: the youth's age, sex, and smoking status, type of outlet, order of purchase (whether it was the first or second attempt to purchase from that outlet), the sex and apparent age of the clerk as judged by the adult, if a sale was completed, if the clerk asked for the youth's age, if ID was requested, if the youth misrepresented his or her age, if the youth pleaded or made up a story, and if the youth presented ID. Data were not collected regarding the purchase of non-tobacco items.

In real life, smokers are “rewarded” for their perseverance by being able to obtain tobacco. To encourage a similar level of engagement for the study, all youths were offered an incentive of a bonus of $1 per pack in addition to their hourly wage for each successful purchase made by the pair. The non-smokers were also provided an incentive to prevent them from becoming discouraged in the event that the smokers were more successful.


The Student's t test was used to compare means for continuous variables. The χ2 test and Fisher's exact test (for small samples) were used to assess the relation between nominal variables. McNemar's test for matched pairs was employed to assess any significant differences between smokers' and non-smokers' ability to purchase tobacco. Because of the large number of variables and the limited sample size, the selection of variables for the full model had to be parsimonious. Variables which, at the bivariate level, were independently associated with successful attempts to purchase cigarettes (at p ⩽ 0.20) were subsequently included in a logistic regression analysis using the SAS system.29 For this analysis, a probability value of p < 0.05 was used as the standard for significance in assessing the odds ratios and confidence intervals of the independent variables.

Upon fitting a final model, the logistic regressions were recomputed adjusting, independently, for individual youths, pair of belonging in the matched pair design, and community, given the lack of independence in the total number of purchases attempted.


All purchase attempts were completed during July and August 2000. Out of 320 purchase attempts, 77% were made by females. Overall, 12.2% of purchase attempts were successful (table 1). Smokers were more successful than non-smokers (McNemar's χ2 for paired data: 11.13, p < 0.001). Bivariate analysis of the 320 purchase attempts revealed that sales rates increased with the age of the youth (table 1), but were not affected by the youth's sex, the clerk's sex, the adult supervisor, or the order of purchase. The average estimated age for the clerks who completed sales was 27.2 years compared to 36.4 years for clerks who refused sales (p < 0.001). Clerks asked the age of the youth in 5.3% of purchase attempts and for ID in 89.1% of attempts. Illegal sales were much more common when ID was not requested (table 1). When ID was requested, smokers completed purchases more often than non-smokers (table 1). Nineteen sales were made to smokers with no request for age or ID compared to nine sales to non-smokers under these conditions.

Table 1

Bivariate analyses of predictors of completed illegal cigarette sales by retail clerks to minors comparing non-smokers following a standard compliance check protocol to smoker's behaving naturally

Regression modelling was utilised to assess the impact of the two protocols while controlling for other variables. The regression model controlled for the sales clerk's sex and estimated age, order of purchase attempt, adult supervisor, whether the teen was asked for his or her age and, independently, whether ID was requested. The models did not include as predictor variables, being able to lie, present ID, tell a story, or plead, as these strategies were only allowed for the smokers. Protocol, the clerk's age, and being asked for ID were the variables that comprised the final regression model. The “smoker” protocol increased the rate of sale nearly sixfold over that for the non-smokers (odds ratio (OR) 5.7, table 2). Completed sales were significantly more likely to occur when the estimated age of the clerk was less than 21 years and when ID was not requested (OR 7.6, table 2). The regression model was adjusted for the community in which the sale attempt occurred. The pair of belonging among the matched pairs of smokers and non-smokers was not included in the final model as this variable has a colinear relation with the community each pair operated in.

Table 2

Logistic regression analysis of predictors of completed illegal cigarette sales by retail clerks to minors comparing non-smokers following a standard compliance check protocol to smoker's behaving naturally


Post-hoc analysis was performed to gain insight into what might account for the greater likelihood of a sale for the “smoker” protocol given that these teens were allowed flexibility in the strategies used to attempt a purchase. Out of 160 purchase attempts by smokers only, teens lied about their age or claimed to have left their ID elsewhere 86% of the time, presented their own ID 8% of the time, and resorted to pleading 6% of the time. Only one smoker possessed ID. Considering only smokers, a sale was more likely when ID was presented (41.7%, 5/12) than when it was not (15.5%, 23/148, p < 0.05). Pleading (n = 9) was never successful. Paradoxically, smokers were sold tobacco 83% (19/23) of the time when they did not lie and 7% (9/137) of the time when they did lie (p < 0.001). The youths did not have to lie if the clerk responded to their initial request. The logistic regression analyses revealed that the adult supervisor accompanying the teen, the order of purchase, the clerk's sex, and whether the teen was asked his or her age were not significant predictors of successful attempts among the smokers. Sample size limitations precluded assessing the protocol variable of “pleading” in the model development, and multicolinearity between telling a story and lying about one's age precluded inclusion of these two variables in the final model. The buyer's sex and age were not included in these regression models because the buyers were too homogeneous on these variables (80% female, 60% age 16 years). Age of the clerk (< 21 years), being asked for ID, and the presentation of ID were the variables composing the final regression model. Smokers were significantly more likely to purchase cigarettes successfully if the clerk was less than 21 years of age, if they were not asked for ID, or if they produced ID (table 3). This final model was adjusted for the community where the purchase attempt occurred though there was found to be no significant effect on the model after controlling for the individual youths (likely due to the colinearity of youth and community).

Table 3

Logistic regression analyses of predictors of completed illegal cigarette sales to minors with the analysis limited to smokers


In unfamiliar stores, smokers were nearly six times more successful at purchasing tobacco than were non-smoking youths following a standard compliance check protocol. While the “smoker” protocol was far more realistic than the standard compliance check protocol, it was unrealistic in one important aspect. Young smokers usually buy tobacco in their own communities and indicate that knowing the person behind the counter is a crucial factor in determining success (JR DiFranza, unpublished data, 2000). The violation rates we obtained using smokers strange to the store personnel are very likely to underestimate the difficulty faced by young smokers living in these same communities. The standard compliance check protocol using non-smoking youths who are strangers to store personnel is not a valid measure of the ability of local youths to purchase tobacco in a setting of strong law enforcement.

The standard protocol differs in several aspects from the natural behaviour of young smokers. It employs non-smoking youths who may not be experienced at purchasing tobacco and who cannot lie about their age or present ID. Smokers were twice as likely to be sold tobacco with no questions asked. This might reflect their greater past experience at purchasing, or greater self confidence. Lack of experience on the part of the non-smokers may have contributed as much to their lack of success as the constraints of the protocol. This is an important issue since some states purposely limit the number of checks youths may do during the annual statewide Synar compliance survey.30State officials in South Carolina allow youths to perform a maximum of nine checks, but in practice, they average only 4.2 checks per youth, almost assuring that all checks are performed by inexperienced youths.31 If the goal is to measure merchant compliance accurately, it would seem prudent to ensure that the buyers are practised and confident.

When our smokers lied about their age or ID in communities with high compliance rates, it had little effect on the measured violation rate because clerks wanted to see an ID. In a small pilot study involving only one 15 year old girl, Hyland and colleagues reported that lying had no impact on her ability to purchase tobacco.32 Since the overall violation rate was 41% in that study, it appears that lying has little impact on measured violation rates whether the law is well enforced or not.32 However, youths from communities with aggressive enforcement indicated that if they were able to buy tobacco once by lying, they could return to the same store day after day to buy from the same clerk (JR DiFranza, unpublished data, 2000). Thus, lying may have little impact on measured compliance rates but it could have a big impact on the long term availability of tobacco to individual youths and their friends.

Radecki has reported that presenting ID with an underage birth date increases the rate of completed sales.33 In this study, only one smoker had an ID. Although the ID proved that she was underage, showing it resulted in a dramatic increase in her ability to purchase over that of the other smokers (OR 27, 95% confidence interval (CI) 3.4 to 212). It is possible that other aspects of this youth's appearance or behaviour could have contributed to this large effect. Perhaps having the ID allowed her to project confidence.

Some authors have reported a positive impact from enforcement on the prevalence of smoking among minors, while others have not.2-6 34-38 Studies in three sites all reported violation rates under 10%, but impacts on the prevalence of smoking were seen in only two.3 5 34-36 38 While measured compliance rates may have been comparable, the ability of neighbourhood youths to purchase tobacco may have been quite different. A crucial question is, what do merchants do when approached by neighbourhood youths who they know are not associated with enforcement? Since merchants can learn to evade being caught in the act, law enforcement might only be effective when it is accompanied by the voluntary cooperation of merchants. It is well established that the voluntary cooperation of merchants in the absence of enforcement is never enough,16 39-41 but voluntary cooperation (that is, merchant buy-in) may be necessary in order for enforcement to impact on smoking prevalence rates.

Two changes to the standard protocol might ensure that the compliance test would be a reasonably valid approximation of what happens when underage smokers attempt to purchase tobacco in stores where they are not known. Youths should produce ID showing that they are underage. Fictitious names and addresses can safeguard confidentiality. If non-smokers are used, they must be experienced and confident. The rate of successful purchases for non-smokers continues to increase even after 60 attempts.24

The fact that the violation rate recorded by our “standard” protocol was within the range reported by local enforcement officials (under 10%) suggests that the protocol we used is equivalent to that used in the enforcement programmes, lending support to the validity of this study. Further research is needed to determine if these results are generalisable to other settings.

The high rate of illegal sales by clerks under 21 years of age provides a strong argument for prohibiting such individuals from selling tobacco.

Since we wanted the smokers to act naturally, we did not dictate the strategies they would use. This resulted in small numbers and poor distribution for a number of predictive variables, limiting the analyses. More research is needed to determine which aspects of the standard protocol contribute to its low sensitivity. By providing the non-smokers with a financial incentive for each pack of cigarettes purchased, we may have biased our study toward a null finding since the standard compliance check does not employ such an incentive. It is typical to train youths before employing them to conduct compliance checks and the fact that our non-smokers did not rehearse may have biased the results to show a greater difference between the two protocols. A logical goal for future research would be to vary different aspects of the protocol one at a time to determine which are important.

The standard compliance check underestimates the willingness of clerks to sell tobacco to strange youths. This may be due to the confidence displayed by underage smokers, their experience, and their ability to produce an ID, even if the ID shows them to be underage. The compliance check does not measure the willingness of clerks to sell to familiar youths. This limits our ability to measure the effectiveness of enforcement efforts, the crucial factor in evaluating the impact of enforcement on the prevalence of tobacco use. More accurate methods for measuring youth access to tobacco from commercial sources are needed. A measure of merchant “buy-in” may help to fill the gap between the standard compliance check and the real world of underage smokers.

What this paper adds

Some have questioned whether merchant compliance checks performed for research or enforcement purposes may be too artificial to reflect accurately the experience of underage smokers. In this study, underage smokers were nearly six times more likely to complete a purchase successfully than were age and sex matched non-smoking youths who followed a standard protocol.

The standard compliance check protocol using non-smoking youths who are strangers to store personnel is not a valid measure of the ability of local youths to purchase tobacco. The currently used methods of assessing merchant compliance provide false reassurance regarding the effectiveness of enforcement on curtailing the illegal sale of tobacco to minors. This finding has important implications for research that would evaluate the effectiveness of tobacco sales enforcement programmes and their impact on rates of tobacco use among youths.


This project was funded by the Substance Abuse Policy Research Program of the Robert Wood Johnson Foundation. The opinions stated here do not necessarily represent those of the sponsors.