Article Text
PDF
Abstract
Purpose Tobacco prevention media campaigns are an important tool to address youth tobacco use. We developed a theory-based perceived message effectiveness (PME) Scale to use when vetting messages for campaigns.
Methods Participants were a national sample of N=623 US adolescents (ages 13–17 years) recruited from a national probability-based panel. In an online experiment, we randomised adolescents to view tobacco prevention ads. All participants viewed an ad on smoking or vaping from the US Food and Drug Administration’s The Real Cost campaign and a control video, in a random order. After ad exposure, we assessed PME using nine candidate items and constructs for convergent and criterion validity analyses. We used confirmatory factor analysis and examined information curves to select the scale items.
Results A brief PME scale with three items (α=0.95) worked equally well for demographically diverse adolescents with different patterns of tobacco use. The Real Cost ads generated higher PME scores than the control videos for both vaping and smoking (convergent validity; p<0.05). Higher PME scores were associated with greater attention, fear, cognitive elaboration and anticipated social interactions (convergent validity; r=0.31–0.66), as well as more negative attitudes toward and lower susceptibility to vaping and smoking (criterion validity; r=−0.14 to −0.37). A single-item PME measure performed similarly to the three-item version.
Conclusions The University of North Carolina PME Scale for Youth is a reliable and valid measure of the potential effectiveness of vaping and smoking prevention ads. Employing PME scales during message development and selection may help youth tobacco prevention campaigns deploy more effective ads.
- advertising and promotion
- electronic nicotine delivery devices
- media
Data availability statement
Data will be available after publication of all papers from this dataset based on reasonable request.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
Introduction
Tobacco use among youth continues to be a major public health issue. Globally, youth smoking prevalence is 11.3% in boys and 6.1% in girls aged 13–15 years,1 while use of other tobacco products (OTPs) (including e-cigarettes) is 11.2% in boys and 7% in girls. While cigarette smoking has been declining in most countries, use of OTPs has remained unchanged or is increasing.1 We define tobacco use to include all tobacco products including e-cigarettes. In the USA, e-cigarettes have been the most commonly used tobacco product among high school-aged adolescents since 2014,2 3 and recent data reveal that 1.72 million high school youth continue to use e-cigarettes.4 While current cigarette smoking is at historical lows in the USA,5 those numbers may begin to rise if youth who begin vaping progress to cigarette smoking.6
Youth tobacco prevention media campaigns reduce the initiation, prevalence and progression of tobacco use.7–10 To slow the vaping epidemic, national campaigns such as the US Food and Drug Administration’s (FDA) The Real Cost are increasingly focusing on vaping prevention.11 The process of developing and selecting ads often involves the use of ratings of perceived message effectiveness (PME).12 PME measures are applied by gathering target audience ratings of the likely impact of persuasive messages on attitudes and behaviours,13 and they predict the impact of smoking cessation ads.14 15
Several PME measures have been developed in the context of adult smoking cessation,15–17 but we are unaware of measures for adolescents or vaping prevention.12 Many social, emotional and physical changes take place during adolescence, making this age group susceptible to initiating tobacco product use.18 Given this, tobacco prevention campaigns often target adolescents,19 and PME measures developed for this context are needed.
Moreover, a growing literature suggests that PME measures tend to involve message or effects perceptions.12 20 Message perceptions are judgements about whether a message promotes further processing that leads to persuasion (eg, ‘This ad is informative’), while effects perceptions are judgements about a message’s potential to change behaviour (eg, ‘This ad discourages me from wanting to vape’).20 Several studies have found that effects perceptions better predict the impact of ads on tobacco-related beliefs and behaviours than do message perceptions.21–23 Therefore, an effects orientation may provide a fruitful basis for PME measures.
In the current study, we sought to develop and preliminarily validate a brief, theory-based effects perception PME measure for youth tobacco prevention in a national sample of US adolescents.
Methods
Participants
Participants were a national probability sample of US adolescents (ages 13–17 years) recruited in September and October of 2020 from the AmeriSpeak panel, a probability-based panel maintained by the National Opinion Research Center (NORC) at the University of Chicago in the USA. NORC randomly selected US households using area probability and address-based sampling, with a known, non-zero probability of selection from the NORC National Sample Frame. For the current study, adolescents were drawn from AmeriSpeak panel households. To address panel attrition due to the COVID-19 pandemic, NORC also invited adolescents aged 13–17 years living in AmeriSpeak panel households who had not yet joined the teen panel to take part in the study. In total, 1351 households had age-eligible children and received information about the study. Parents from 1002 households (74% of those eligible) provided informed consent, and 624 adolescents assented and completed the survey (62% of households whose parents consented; 46% of all eligible households). One participant had extensive missing data and was excluded from analyses, resulting in N=623.
Study design and procedures
To assess validity for the PME Scale, we randomised adolescents to view either vaping or smoking prevention 30-second video ads (online supplemental figure 1). All participants viewed an ad from the US FDA’s The Real Cost campaign and a neutral control ad, in a random order. For The Real Cost, participants saw one of six ads; for control, participants viewed an ad we developed with neutral information about vaping or smoking. To create control ads, we adapted content from Wikipedia and other sources, such as product definitions and production methods, using parallel language for cigarettes and e-cigarettes. The control ads featured black text on a white screen with audio narration. After watching the first ad (twice to maximise exposure), participants answered PME and message reactions items, followed by vaping and smoking outcomes. They then watched the second ad—again twice—and answered PME and message reactions items.
Supplemental material
Participants received a $12 cash equivalent incentive through the NORC panel. We preregistered our study on aspredicted.org (https://aspredicted.org/gs8fr.pdf).
PME measure
We conceptualised our youth PME Scale as an ‘effects perception’ measure and developed items in three theoretically informed domains based on the Reasoned Action Approach,24 with parallel items for vaping and cigarette smoking. We selected these domains because the goal of tobacco prevention messages is typically to affect beliefs, attitudes and motivation, with the goal of changing behaviour.25 26 The first domain was beliefs about the consequences of tobacco use, with a focus on health harm and addiction consequences.27 The second domain was attitudes about the behaviour, with a focus on unfavourable perceptions of tobacco use.24 The final domain was motivation, with a focus on intentions to avoid using tobacco.24 We developed candidate items in each of the theoretical domains informed by the PME literature,12 including items from our previously developed PME Scale.20 When developing items, we focused on accessible language, linguistic appropriateness and theoretical relevance.
Through an iterative process of drafting, revising and winnowing items, we developed a set of 11 candidate items across the three domains. Next, we conducted two rounds of cognitive interviews via telephone with 10 adolescents a piece (N=20 total),28 focusing on readability, comprehension and item meaning.29 Based on findings from the cognitive interviews, we refined our items and selected nine items for use in the present study (table 1; see Kurtzman et al 28). Prior PME measures have shown positive skew, limiting the ability to distinguish between messages at higher levels of PME,20 so we adopted a unipolar response scale (‘not at all’ to ‘a great deal’) rather than a bipolar response scale (‘strongly disagree’ to ‘strongly agree’) in an attempt to reduce this skew; this new response scale was well understood by adolescents in the cognitive interviews.
Frequencies of the nine PME items in the national survey (calibration sample, N=623)
Other measures
Message reactions
To support analyses of convergent validity (the extent to which PME is associated with variables that it should theoretically relate to), the survey assessed several constructs from the Tobacco Warnings Model,30 with hypothesised positive correlations with PME (except reactance for which negative correlations were expected). The survey assessed attention with the item, ‘How much does this ad grab your attention?’; negative affect with the item, ‘How much does this ad make you feel scared?’ and cognitive elaboration with the item, ‘How much does this ad make you think about reasons for not vaping (smoking cigarettes)?’. The survey also assessed reactance with the item, ‘How much does this ad annoy you?’ and avoidance with the item, ‘How much does this ad make you want to look away?’. The survey assessed anticipated social interactions with three items, including ‘How much does this ad make you want to talk to your friends about the harms of vaping (smoking)?’. All message reaction items had a 5-point scale that ranged from ‘Not at all’ (coded as 1) to ‘A great deal’ (5).
Psychosocial outcomes
The survey additionally assessed several outcomes24 31 for purposes of criterion validity (the extent to which PME predicts a relevant outcome). The survey assessed perceived likelihood of harm with the item, ‘If you regularly vaped (smoked cigarettes), what is the chance that you would one day get vaping-related (smoking-related) health problems?’ on a 5-point scale ranging from no chance (coded as 1) to certain (5). The survey assessed attitudes about vaping/smoking with the item, ‘Do you think vaping (smoking cigarettes) is….’ on a 5-point scale ranging from very bad (1) to very good (5). Finally, the survey assessed susceptibility to vaping/smoking using a three-item susceptibility scale: ‘Do you think you might use an e-cigarette or vape soon?’, ‘Do you think you will use an e-cigarette or vape in the next year’, and ‘If one of your best friends were to offer you an e-cigarette or vape (cigarette), would you use (smoke) it?’.31 Responses were on a 4-point scale from definitely not (1) to definitely yes (4).
Vaping/smoking status
The survey assessed whether youth had vaped or smoked cigarettes in the past 30 days, and those who had were classified as a current user. If they had used the tobacco product before, but not in the past 30 days, we assessed whether they thought they would use the product in the future, on a 4-point scale ranging from definitely not (1) to definitely yes (4).31 If they answered anything other than ‘definitely not’, we classified them as at risk of vaping/smoking. For youth who had never used the tobacco product at all, the survey assessed whether they had ever been curious about using the tobacco product,32 and also if they thought they would use the tobacco product in the future.31 If they answered anything other than ‘definitely not’ to both questions, we classified them as at risk of vaping/smoking. We classified all other adolescents as not at risk of vaping/smoking.
Demographics and OTP use
The survey assessed age, gender, race, ethnicity, parents’ education, sexual attraction and use of OTPs in the past 30 days.
Data analysis
Our overarching goal was to develop a brief youth PME Scale using the following criteria: (1) the scale should include one item from each theoretical domain for adequate coverage of the PME construct; (2) the scale should be unidimensional; (3) items should work equally well across demographic subgroups; (4) the scale should not have ceiling or floor effects; and (5) the scale should have good convergent and criterion validity.
To achieve these goals, we created a calibration sample and a validation sample using a random number generator. Because each participant evaluated two ads, half the sample contributed their first PME rating to the calibration sample and their second rating to the validation sample. The other half of the sample did the opposite. This process resulted into two datasets, each with N=623 independent observations.
To evaluate dimensionality of the PME measure, we conducted an exploratory factor analysis (EFA) on the calibration sample using Mplus V.8.4. We drew from the EFA results to construct a confirmatory factor analysis (CFA), evaluating model fit with a unidimensional model with all nine items. Fit indices were the confirmatory fit index (CFI; adequate fit >0.95), Tucker Lewis Index (TLI; adequate fit >0.95) and root mean square error of approximation (RMSEA; adequate fit <0.08). To evaluate potential floor and ceiling effects, we calculated item information from the item intercept and factor loading.
To evaluate suitability of the PME measure for diverse populations, we investigated measurement non-invariance (also called differential item functioning) by regressing factor loadings and item intercepts on covariates of interest33 34 to determine whether items operated differently across several covariates. We used the MNLFA package in R35 to test each item iteratively while using the other items as anchors.
To pare down the scale, we removed theoretically and empirically redundant items that had the lowest levels of item information, as well as items exhibiting measurement non-invariance. We also evaluated the performance of a ‘best’ item to use if a study only has room for a single PME item.
Finally, we conducted analyses on the validation sample using a structural equation model, regressing the latent PME factor (the final three-item scale) on demographic and tobacco use variables, controlling for ad product type and type of ad. We repeated this analysis with the single ‘discourage’ item in a proportional odds regression model with a probit link function estimated using Mplus V.8.4.
Results
Participant characteristics
The mean age of adolescents was 15 years. Most participants reported being female (53%) and white (65%), and roughly one-fifth (19%) identified as Hispanic. Nearly half (49%) of adolescents had a parent with a bachelor’s degree or higher. Fourteen percent of the sample were current vapers (8% were smokers), while 47% were at risk of vaping (33% for smoking) and 39% were not at risk of vaping (59% for smoking). Use of OTPs was low (1%–6%). Thirty-six percent reported family member tobacco use in the home (online supplemental table 1).
Psychometric analyses with the calibration sample
About one-third of participants answered ‘a great deal’ to PME items, the high end of the scale (table 2). This somewhat limited our ability to discriminate scores at the high end of PME. Interitem correlations were all high (polychoric r=0.89–0.90), a strong indication of unidimensionality and redundancy among the full set of nine items.
Correlations between the UNC PME Scale for Youth and convergent and criterion validity outcomes (validation sample, N=623)
Factor analyses
In the EFA, almost all items loaded onto one factor. The largest eigenvalue was 8.3, followed by 0.2 (suggesting a single factor), but likelihood ratio tests suggested that the two-factor model fit better than the one-factor model, and the three-factor model fit better than the two-factor model. The one-factor model had a CFI-TLI of 1.00 (indicating excellent fit), but an RMSEA of 0.10 (indicating poor fit). For the two-factor model, CFI-TLI was 1.00 (excellent fit) and RMSEA was 0.07 (only minimally acceptable); while for the three-factor model, CFI-TLI was 1.00 (excellent fit) and RMSEA was 0.03 (good). Factor loadings for the three-factor model appear in online supplemental table 2.
In the three-factor model, all items except ‘convince’ loaded well on the first factor. ‘Convince’ was by itself loading on a second factor, with no loading on the first factor. ‘Harm’, ‘negative effects’, ‘worry’ and ‘unpleasant’ all had small (ranging from 0.06 to 0.25) but statistically significant loadings on the third factor, suggesting that they shared some common variance. Because ‘convince’ did not load with the other eight items, we dropped it from the CFA.
In the CFA, we tested a bifactor model for the remaining eight items to determine the extent to which each of the four cross-loading items (‘harm’, ‘negative effects’, ‘worry’ and ‘unpleasant’) reflects the PME factor of interest versus the orthogonal ‘nuisance’ factor, thereby isolating the common item variance due to PME alone. The bifactor model fit well (CFI-TLI=1.00; RMSEA=0.02). All items loaded between 0.94 and 0.97 on the first factor, which we now term PME. On the second factor, the four items identified above had small but statistically significant loadings (0.05–0.26).
Item information curves showed that all items contributed information along a similar range of PME, from −1.5 SD below the mean to +1 SD above the mean. ‘Worry’, ‘bad idea’ and ‘discourage’ exhibited the highest information, and each of these items represented a different theoretical domain (online supplemental figure 2). Thus, these items became candidates for the scale. We used the ‘discourage’ item in the single-item analyses, because (1) we viewed it as theoretically closest to behaviour; (2) it had high information; and (3) it has been used successfully in prior studies of tobacco and sugary drink warnings research.36
When testing for measurement non-invariance, we found that participants viewing smoking ads were more likely to endorse the ‘unpleasant’ item, and that current smokers were less likely to endorse the ‘avoid’ item. Given that each of the three candidate items for the scale exhibited measurement invariance, meaning that measurement characteristics of these items were stable across all of the demographic variables that we tested, we selected ‘bad idea’, ‘discourage’ and ‘worry’ for the three-item scale.
The three-item scale is just identified, meaning that fit indices are not informative. However, all factor loadings were statistically significant and high, ranging from 0.93 (for ‘worry’) to 0.97 (for ‘bad idea’) (figure 1). Consistent with these factor loadings, item information remained highest for ‘bad idea’ and ‘discourage’ and slightly lower for ‘worry’. Coefficient alpha for the three-item scale was very high (alpha=0.95), indicating the scale is reliable.
Confirmatory factor analysis of the UNC PME Scale for Youth (factor loadings with SEs; calibration sample, n=623). UNC, University of North Carolina.
Validity analyses with the validation sample
Youth rated ads from The Real Cost campaign as more effective than neutral control ads (p<0.05; see figure 2). The vaping The Real Cost condition yielded PME scores that were 1.43 (95% CI: 0.92 to 1.94) units higher than control, while the smoking The Real Cost condition yielded PME scores that were 1.10 (95% CI: 0.57 to 1.63) units higher than control. This demonstrates convergent validity of the scale. PME was also positively associated with attention, fear, cognitive elaboration and anticipated social interactions (all p<0.05), further demonstrating convergent validity (table 2). PME was moderately associated with avoidance (for smoking ads only, p<0.05) and not associated with reactance.
Impact of The Real Cost and control ads on three-item UNC PME Scale for Youth (scores with ±1 SE bars; validation sample, n=623). PME, perceived message effectiveness; UNC, University of North Carolina.
PME was also moderately associated with perceived likelihood of harm (for vaping only) and negatively associated with attitudes toward vaping and smoking and susceptibility to vaping and smoking, demonstrating criterion validity (all p<0.05). All correlations were in the expected direction. The one-item PME measure performed similarly to the three-item measure for most validity constructs.
Predictors of three-item and one-item PME scores
Patterns of statistical significance and magnitude of effects were remarkably consistent for the three-item and one-item PME Scales (table 3). Adolescents whose parents had less than 4 years of college reported higher PME than adolescents whose parents had at least a college degree (b=0.53 for the three-item scale and b=0.61 for the single-item scale), and heterosexual adolescents reported higher PME than youth attracted to others of the same sex (b=0.41 and b=0.36). Adolescents who vaped (b=−0.36 and b=−0.41) or who smoked (b=−0.70 and b=−0.61) had lower PME scores compared with adolescents who were not at risk of tobacco use. Adolescents who were at risk of smoking had significantly lower PME scores than adolescents who were not at risk of smoking (b=−0.39 and b=−0.36).
Multivariable correlates of UNC PME Scale for Youth (validation sample, N=623)
Discussion and conclusion
Youth tobacco prevention media campaigns are a key tool to reduce tobacco use, but the field has lacked an effects-oriented youth PME measure for vetting messages for campaigns. In this study, we developed and preliminarily validated such a measure—the University of North Carolina (UNC) PME Scale for Youth, which assesses the potential effectiveness of tobacco prevention messages for youth (see online supplemental table 3 for the final scale).
The UNC PME Scale for Youth exhibited experimental evidence of convergent validity with both smoking and vaping prevention ads, distinguishing between ads from the US FDA’s The Real Cost campaign and control ads. The scale demonstrated convergent validity through correlations with a series of message processing variables,30 including attention, negative affect, cognitive elaboration and anticipated social interactions. The scale also showed criterion validity through correlations with perceived likelihood of harm (for vaping only), attitudes and susceptibility, consistent with prior work.13 22 37 This begins to build support for the validity of this new scale across tobacco products.
This work grew out of our work developing the original UNC PME Scale20 and synthesising existing measurement12 and validation14 of PME research. The original UNC Scale was developed with adults in the context of cigarette health warnings, and has since been applied to several health contexts,36 38 mostly with adults but also with youth.20 In health warning studies that include adults and youth, a single scale may be desired, and for those applications the appropriate choice may be the original UNC PME Scale. In applications to youth tobacco prevention campaigns, however, the UNC PME Scale for Youth is likely to be the optimal choice. Additional work should also examine whether our new measure is applicable to tobacco prevention and control campaigns for young adults, as well as adapting it for use with youth in other behavioural areas beyond tobacco use.
Finally, since PME measures are often used to evaluate many messages,39 40 brevity is important. Given this, both the original UNC PME Scale and the UNC PME Scale for Youth were designed to be brief. Given this, we also tested the single ‘discouragement’ item compared with the full three-item PME measure, finding little loss of information. Thus, our findings indicate that in contexts where researchers need to reduce response burden, they can feel confident using the single ‘discouragement’ item.
Strengths of our study include the iterative development process of a theory-based measure, cognitive testing of items, use of a national probability sample and item response theory analyses. Our study was limited in that we only tested the PME measure with vaping and smoking ads; a future study could adapt this measure for OTPs. Also, this study was unable to examine whether this PME measure foretells the impact of tobacco prevention ads on adolescents’ beliefs and behaviour over time. Future studies should rigorously evaluate the extent to which PME ratings correspond with actual message impact over time.12 14
In conclusion, this study developed the UNC PME Scale for Youth, a measure that assesses the potential effectiveness of tobacco prevention ads. The scale can be used to help select messages for vaping and other tobacco prevention campaigns, and can also be used to assess receptivity to messages that have been deployed.41 This measure can further be used in validation studies examining the extent to which PME foretells ad impact. Improving PME measurement may lead to better use of campaign resources and ultimately more impact in preventing tobacco use and its associated consequences among youth.
What this paper adds
What is already known on this subject
Tobacco prevention campaigns commonly select messages based on perceived message effectiveness (PME) ratings.
What important gaps in knowledge exist on this topic
Researchers have yet to systematically develop and validate PME measures for youth or for vaping prevention.
What this paper adds
The University of North Carolina PME Scale for Youth—a new brief three-item measure—worked equally well for demographically diverse adolescents with different patterns of tobacco use.
The scale distinguished between the USA’s The Real Cost campaign ads and neutral control ads for both vaping and smoking.
Data availability statement
Data will be available after publication of all papers from this dataset based on reasonable request.
Ethics statements
Patient consent for publication
Ethics approval
This study involves human participants and was approved by the University of North Carolina Institutional Review Board. Participants’ parents gave informed consent and youth gave assent to participate in the study before taking part.
Acknowledgments
The authors would like to thank UNC Connected Health for Applications & Interventions (CHAI) core for their cognitive interview work and NORC at the University of Chicago for their data collection efforts. We also thank Xiaoquan Zhao and Emily Brennan for their contributions to this work.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Contributors SMN, NTB, MGH, JMS and NG designed the study. NG analysed the data. SMN drafted the first version of the manuscript along with NG. SMN is guarantor and acccepts full responsiblity for the work. All authors contributed to the research reported in the paper, critical review and revision of the paper.
Funding This project was supported by grant number R01CA246600 from the National Cancer Institute and Food and Drug Administration (FDA) Center for Tobacco Products (CTP). MGH’s work on this paper was supported by the National Heart, Lung, and Blood Institute of the National Institutes of Health (NIH) under grant number K01HL147713.
Disclaimer The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the FDA.
Competing interests SMN has served as a paid expert witness in litigation aganist tobacco and e-cigarette companies.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.