Sample

The target population for the PATH-RV Study (like that for the main PATH Study) was the civilian household population in the USA age 12 and older. The samples for both studies were stratified, multistage area probability samples, with metropolitan areas and counties selected in the first stage, individual blocks or adjoining blocks selected in the second stage, individual addresses in the third stage and sample persons in the final stage. The PATH-RV Study sample was selected in a subsample of the first-stage samples areas for the PATH Study. Online supplementary appendix A provides the details of the PATH-RV sample design.

The PATH-RV Study sample included 9782 sample addressees, which were mailed a short screening questionnaire to identify adult (18 and older) tobacco users and non-users and youths (12–17 years old). A total of 2296 households returned questionnaires and we selected 865 adults and 266 youths for the PATH-RV sample. In households where a youth was selected, we also randomly sampled one of the adults. In households with more than one youth, one of them was selected at random.

Data collection

Data collection took place in two phases. The first was the mail screening effort, which consisted of up to six mailings to sample addresses: an advance letter, an initial survey package with a cover letter, screening questionnaire and US$5 cash incentive (later reduced to US$2); a thank you/reminder postcard; a replacement survey package (with a replacement questionnaire) and a final reminder postcard (later increased to two reminders). We received 2296 completed screening questionnaires; at another 643 addresses, the mailings were returned as undeliverable. The overall response rate to the screening component of the study was 25.1% (American Association for Public Opinion Research response rate 3).25

Westat field interviewers, all experienced PATH Study interviewers, carried out the field work. The PATH-RV Study training covered the components of the study that differed from the main PATH Study, such as the request for a reinterview. The training consisted of a home study lasting about 4 hours and a 1-hour group session via WebEx. A total of 68 interviewers conducted the PATH-RV field data collection.

This consisted of an initial interview and reinterview done 6–24 days later. Both were done using ACASI, the same mode used in the main PATH Study. In ACASI, the computer displays the questions to the respondents on screen and also plays them aloud to the respondents via earphones. As in the main PATH Study, a text-to-speech synthesised voice was used to generate the audio version of the questions. Both interviews used the main PATH Study wave 4 questionnaire. The median interview lengths for the initial interviews were 65.7 min for the adults and 41.6 min for the youths; the median reinterview lengths were 67.4 min for the adults and 38.8 min for youths.

As in the main PATH Study, adults were offered US$35 for completing each interview and youths, US$25. Both youths and adults were offered US$10 to provide saliva samples. Prior to contacting sample youths, interviewers first obtained parental consent.

With a few exceptions, the protocol was the same in the reinterview done 6–24 days later as in the initial interview. The reinterview questionnaire included some additional items after the regular PATH Study questions. Among the items added were questions asking the reasons for discrepant answers for selected items. The possible reasons for discrepancies included true change, misunderstanding the question, memory problems, inattention and reluctance to answer truthfully. These discrepancy probes were administered at the end of the reinterview; the programme had a record of the respondent’s answers in the earlier interview and could detect discrepancies in real time.

After the reinterview, respondents were asked to provide saliva samples. Respondents who agreed provided a saliva sample using the Alere iScreen screening device, which detects cotinine for up to 4 days after tobacco use. The device includes a saliva-absorbing sponge; respondents were to keep the sponge in their mouths until it had completely softened. The interviewer then placed a cap over the sponge, collapsing it and initiated the test. Three results were possible—positive (cotinine present), negative or invalid (inconclusive results, reflecting insufficient saliva or other problems).ⁱ For adult respondents who reported using any tobacco products in the second interview, the interviewer also asked to photograph the products. For the 110 respondents who agreed, the interviewer photographed the product(s) using the computer’s camera. The photographs were subsequently coded for the product type (eg, cigarettes) and brand (eg, Newport); we compared these to the products and brands reported in the reinterview.

Table 1 shows the outcomes of the field work, including the number of sampled persons who completed the PATH-RV interviews and the number who provided saliva samples. Overall, 46.3% of the sample members completed the two interviews; 89.5% of those also provided a saliva sample.

Comparison to main PATH Study

A central goal for the PATH-RV Study design was to reproduce the essential features of the main PATH Study data collection. We used the same questionnaire, mode of data collection, software to administer the questions and interviewers. Table 2 summarises the key features of the two studies.

sis focuses on four questions: (1) How reliable are the PATH Study wave 4 data? (2) How do the reliability estimates for these questions compare to those for similar NSDUH questions? (3) When discrepancies over time were found, what are the reasons for them? and (4) Do the interview responses agree with the saliva test results and photograph data? The results here are unweighted, since we are not making population estimates.

Reliability of the interview data

We calculated reliability estimates for every item for which at least 100 respondents answered the question in both interviews—a total of 447 questions from the adult questionnaire and for 229 from the youth questionnaire. Because of skip patterns in the questionnaire, some questions were administered to relatively few respondents.

Table 3 presents the results for nine questions asking about current use of various tobacco products. For the adults, the average kappa value is 0.72; for the youths, the average kappa is 0.79. According to Landis and Koch,26 this constitutes ‘substantial’ agreement. The kappa values are low for some products because very few respondents reported using them. For both age groups, the GDRs indicate that over 95% gave identical answers in both interviews. Results for past year use and for lifetime use are similarly high (see below).

The reliability estimates are lower for the questions asking about the respondent’s awareness of various tobacco products (table 4), ranging from 0.32 to 0.64 for the adults and from 0.25 to 0.57 for the youth. In some cases, the answers may differ because respondents learnt about the product in the first interview. This appears to be the case for snus and dissolvable tobacco, where most of the discrepancies involve respondents reporting they were unaware of the product in the first interview but aware of it in the reinterview. Since these are both generally less well known than the other products, it is likely that respondents would have become aware of them because of the first interview. For the adults, 40 of the 72 discrepant answers for snus and 39 of 46 discrepant answers for dissolvables follow this pattern; for the youth, 22 of the 33 discrepant answers for snus and 18 of 21 discrepant answers for dissolvables follow this pattern. The discrepant results for the remaining tobacco products show no clear pattern.

Table 5 presents mean kappas and GDRs for all the questions we analysed, by four different response formats—dichotomous items (mostly yes-no questions), categorical items with more than two response options, ordinal rating scales and items asking for numerical responses. There were only five numerical items for the youth, and they are omitted from the table.

For the categorical and ordinal questions, we examine kappas and GDRs based on exact agreement across interviews and also based on approximate agreement (answers within one category of the earlier response). We refer to the latter statistics as generalised kappas and GDRs. We also present mean weighted kappas, which take the size of the discrepancy into account. The mean kappas are in the 60s (‘substantial agreement,’ according to Landis and Koch,26) and when respondents gave answers that did not agree exactly, they were often within one response category, producing much higher generalised kappas and much lower generalised GDRs. For the numerical questions, we examined the Pearson correlation of answers across two interviews as the measure of reliability.

Comparison with NSDUH reliability estimates

How do the estimates from the PATH-RV Study compare with similar estimates from the 2006 NSDUH Reliability Study? As can be seen from table 6, they are quite similar. Although the question wordings are not necessarily identical in the two surveys, they are generally similar and both studies use ACASI as the mode of interviewing. The kappa values are close for the two surveys and there is no clear pattern of differences across the two—sometimes the PATH Study reliability is higher, sometimes the NSDUH reliability is higher and sometimes the two estimates are identical.

Reasons for discrepancies

As already noted, when the answers in the two interviews were discrepant, they were often close—within a scale point. Table 7 illustrates this point, with reliability estimates for five ratings of the respondent’s health and quality of life. The first four of the variables were assessed on five-point scales, ranging from ‘excellent’ to ‘poor.’ The final variable was also assessed on a five-point scale that ranged from ‘extremely satisfied’ to ‘not at all satisfied’. The generalised kappas and GDRs indicate that there is almost perfect agreement across interviews when this more liberal standard of agreement is applied.

The PATH-RV reinterview questionnaire also probed respondents about why they gave different answers in the two interviews. Items subject to discrepancy probes include lifetime tobacco use, current tobacco use, tobacco products and brands, and health and risks. Table 8 shows the distribution of responses across all the discrepancy probes. (Different items were probed for the youth and adults, so the results for the two groups are not comparable.) True change between the two interviews accounts for a substantial proportion of the discrepancies, though for a higher proportion of the discrepancies for the youth respondents (44%) than for the adults (11%). Lapses of attention also account for many of the discrepancies. Respondents attributed very few of their discrepant answers (less than 2% in both samples) to discomfort in answering the questions truthfully.

Validity of the answers

That the answers were consistent (or at least close) over time does not guarantee that they were accurate. Our study included two validity checks. First, we compared respondents’ answers to the tobacco use questions to the saliva test results. The survey results reflect respondents’ answers to questions about current use of several different types of tobacco products (eg, ‘Do you now smoke cigarettes every day, some days or not at all?’). Respondents were classified as current users if they reported using any tobacco product every day or some days. Table 9 shows that the survey data and saliva test results agree for 87.5% of the respondents (kappa=0.72). Of the 15 respondents who reported they were tobacco users in the second interview but tested negative, 12 were occasional users (saying they used one or more tobacco products ‘some days’).

Second, the reports of the adult respondents about the types of tobacco products they were currently using (and which particular brands) were compared with the products identified by coders from the photographs interviewers took. At the product level, the photographs matched the survey reports 98.1% of the time. If the respondent reported using cigarettes, the photograph almost always showed cigarettes (in 85 out of 86 instances), and, if they reported using some other product, the photograph almost always showed that other product (19 out of 20 instances). There were more discrepancies at the level of brands. The brand reported by the respondents agreed with the brand identified in the photographs 87.5% of the time. (The survey item asking about brand only lists brands rather than variety or sub-brand. Thus, if a respondent reported ‘Marlboro’ and the photograph showed Marlboro Lite, it was considered a match.)