NOT PEER REVIEWED
These arguments by Pesko and Friedman cannot undo the central flaw in the Friedman paper. We are surprised that Pesko and Friedman continue to argue that Friedman’s analysis of the YRBSS fall data as “after” data is valid despite the Friedman paper defining the exposure variable as follows: “A binary exposure variable captured whether a complete ban on flavoured tobacco product sales was in effect in the respondent’s district on January 1 of the survey year.”[1] If Friedman had intended to treat the period immediately after July 21 2018 as the “after” period, why had she not selected July 21 of each year as the cut-off date for indicating exposure to the policy effects? It seems apparent that Friedman chose the January 1, 2019 as the cut-off for “after” data because she knew this was the enforcement date and she assumed wrongly that the YRBSS data were collected after January 1, 2019. This is evident in her own response[2] to a critique[3] of her paper as we already noted in our previous response.[4]
Friedman states that “the official/legislated effective date are used to ensure that resulting estimates capture unconfounded responses to the policy change.” Again, if this approach made sense in the specific San Francisco case, why did Friedman use January 1, 2019 in her paper? Perhaps because it simply doesn’t make sense to attribute a policy’s effects before the policy is actually implemented. Similarly, the use of enforcement date rather than...
NOT PEER REVIEWED
These arguments by Pesko and Friedman cannot undo the central flaw in the Friedman paper. We are surprised that Pesko and Friedman continue to argue that Friedman’s analysis of the YRBSS fall data as “after” data is valid despite the Friedman paper defining the exposure variable as follows: “A binary exposure variable captured whether a complete ban on flavoured tobacco product sales was in effect in the respondent’s district on January 1 of the survey year.”[1] If Friedman had intended to treat the period immediately after July 21 2018 as the “after” period, why had she not selected July 21 of each year as the cut-off date for indicating exposure to the policy effects? It seems apparent that Friedman chose the January 1, 2019 as the cut-off for “after” data because she knew this was the enforcement date and she assumed wrongly that the YRBSS data were collected after January 1, 2019. This is evident in her own response[2] to a critique[3] of her paper as we already noted in our previous response.[4]
Friedman states that “the official/legislated effective date are used to ensure that resulting estimates capture unconfounded responses to the policy change.” Again, if this approach made sense in the specific San Francisco case, why did Friedman use January 1, 2019 in her paper? Perhaps because it simply doesn’t make sense to attribute a policy’s effects before the policy is actually implemented. Similarly, the use of enforcement date rather than effective date is not at all as unusual as Pesko claims. Pesko and Friedman’s suggestion to use effective date post hoc simply does not make any logical sense in the San Francisco case, where there was an explicit and highly publicised period of non-enforcement as well as documented non-compliance with the policy through the period of survey administration. In fact, all the existing papers on the San Francisco flavour ban,[5–8] including the Friedman paper,[1] have used the January 1, 2019 enforcement date as the cut-off date for evaluating the policy implementation effects.
Friedman rightly points out that the San Francisco Department of Public Health didn’t even begin compliance inspections until December 3rd, 2018. The YRBSS survey was already nearly complete (fielded between November 5th and December 14th, 2018) at that time. In addition, the current smoking question assesses smoking in the past 30 days, meaning that all of the survey respondents would be reporting on their smoking behaviour for a preceding period that encompasses a time before compliance checks began. When compliance checks began in December, only 17% of retailers were found to be compliant with the flavour ban, likely because they were explicitly instructed that there would be no penalties until January 1, 2019. These facts mean that youths’ retail purchase access would not have changed appreciably at that time. Her conclusion in her paper that “reducing access to flavoured electronic nicotine delivery systems may motivate youths who would otherwise vape to substitute smoking”[1] is inconsistent with the fact that e-cigarettes were still widely available in San Francisco in the fall of 2018.
Pesko and Friedman cite Gammon et al. (2021) showing reduced e-cigarette sales[5] to argue that Friedman’s analysis is still valid because the law may have led to a decrease in youth’s demand for e-cigarettes before the enforcement date. In truth, the vaping rate went from 7.1% to 16% in San Francisco between 2017 and 2019. We note that the Friedman paper omitted reporting youth vaping prevalence,[1] stating that “Recent vaping was not considered because of likely confounding. California legalised recreational marijuana use the same year San Francisco’s flavour ban went into effect; in addition, the YRBSS’s vaping questions did not distinguish vaping nicotine vs marijuana.” The decision to not control for vaping in the Friedman analysis is not justified. Friedman wrote in her response[2] to three critiques[3,9,10] of the original paper that the reason was potential misclassification of marijuana vaping due to California’s legalisation of recreational marijuana because the YRBSS questions do not specify the substance being vaped. Marijuana exclusive vapers account for only about 1% of the youth population, making this an inappropriate reason to not control for significant differential changes in vaping over time in different cities.[11–13] For example, vaping rates went down in Oakland after the flavour restriction but were up significantly in the 2018 pre-enforcement period in San Francisco. Initiation of vaping nicotine has been associated with higher rates of subsequent use of cigarettes among adolescents.[14,15] Higher rates of vaping nicotine e-cigarettes may also have been the impetus for passage of the San Francisco flavour ban, making vaping an important confounder. Taken together, these facts make uncontrolled confounders a likely explanation to cigarette use differences across locations and therefore decreases the possibility that the cigarette smoking rate went up due to an unenforced flavour ban.
Pesko and Friedman did not mention that Gammon et al. (2021) reported the predicted flavoured nicotine e-cigarette sales in San Francisco increased from 3439 units per week pre-policy (July 2015-July 2018) to 5906 units per week in the effective period (July-December 2018) and only declined after the enforcement period (January-December 2019) to 16 units per week (Table 1 in their article).[5] Clearly, flavoured e-cigarettes were still widely available in the marketplace during the effective but non-enforced period and in fact, more flavoured e-cigarettes were sold during the effective period than prior to the policy. Furthermore, Friedman also did not mention that Gammon et al. (2021) reported that cigarette sales declined post flavour ban.[5] Predicted total cigarette sales in San Francisco declined from 83424 units per week pre-policy (July 2015-July 2018) to 77370 units per week in the effective period (July-December 2018) and further declined after the enforcement period (January-December 2019) to 64220 units per week (Table 1). [5] This pattern is therefore inconsistent with Friedman’s 2021 paper conclusion that “reducing access to flavoured electronic nicotine delivery systems may motivate youths who would otherwise vape to substitute smoking” in the fall of 2018. The fact is average weekly flavoured e-cigarette sales increased while total cigarette sales decreased in San Francisco between July-December 2018 compared to pre-policy period. [5] The substitution explanation falls apart. Pesko and Friedman cannot selectively use data to have it both ways.
As we described in our paper, after Oakland implemented a convenience store flavoured tobacco sales restriction in July 2018, high school youth vaping declined from 11.2% to 8.0% (p=0.04)[16] and smoking declined from 4.4% to 2.4% (p=0.02)[17] between 2017 and 2019. Our description that vaping and cigarette use prevalence declined was accurate. Upon reviewing the YRBSS data from the CDC, the Oakland data does in fact represent a statistically significant drop in vaping and smoking rate from 2017 to 2019. Friedman objects to our use of the Oakland (neighboring city to San Francisco) data as a comparison because Oakland’s law was less comprehensive than San Francisco’s. We respectfully disagree with Friedman’s objection. The Oakland law that drastically limited youth access to flavoured tobacco products in that city certainly informs the San Francisco case. The idea that the decline in cigarette smoking prevalence after the flavour ban in Oakland was less than the decline of cigarette smoking elsewhere is disproven by the fact that there was a greater drop in current smoking rate in Oakland from 2017 to 2019 (46% decline from 4.4% to 2.4%) compared to the average decrease nationally across the United States (32% decline from 8.8% to 6.0%) based on YRBSS data.[18]
Friedman offered several post hoc explanations for why youth cigarette smoking might increase following a flavour ban. She offers no data from San Francisco to support market responses following the SF flavour ban, nor does she provide data that SF youth had switched to using flavour accessories. These scenarios also assume that flavoured tobacco products were no longer available at the time of the SF YRBSS data collection, but we know products were still largely available as of December 2018 in 83% of the retailers. It is historically inaccurate for Friedman to suggest that the outbreak of EVALI had any bearing on potentially reducing people’s willingness to buy vaping products from informal sellers in 2018 because this outbreak occurred in the fall of 2019, one year after the SF YRBSS data were collected.
Our description about receiving the YRBSS survey collection date through an inquiry from the CDC was accurate.[19] The CDC informed us that the YRBSS in San Francisco was conducted in the fall of 2018 and we used this information in our paper. We wrote to the San Francisco School District to confirm these dates as did Dr. Friedman.
Liber’s points about partial compliance rates are refuted by the availability of flavoured products during the survey administration period and are addressed by our above response. We thank him for agreeing that this case highlights the need for including more precise date of data collection identifiers in publicly available data sets. Given the significance and potential impact of these analyses for public health policy, it behooves all users of publicly available data to pay close attention to dates of data collection in relation to policy effective/enforcement dates when analyzing this information, to seek confirmation if there is any doubt, and not make assumptions about the dates. In this case the dates of the 2019 YRBSS administration ranged widely from fall of 2018 (SF) to fall of 2019 (NYC).[19]
An important benefit of flavour ban legislation is that flavoured combustible tobacco use goes down.[7] The use rates of flavoured combustible little cigars and cigarillos are similar or exceed the combustible cigarette use rate among youth in San Francisco, [11] making flavour bans an important tool in decreasing overall youth combustible tobacco rates.
The results of the 2019-2020 California Student Tobacco Survey, which was conducted after the enforcement of the flavour ban showed that the prevalence of cigarette smoking among San Francisco high schoolers was 1.6% (compared with 4.7% based on the San Francisco 2017 pre-ban YRBSS data).[11] After the enforcement of the flavour ban, we now see historically low smoking rates in San Francisco. This data from the time after the flavour ban was actually implemented by retailers further calls into question the conclusion of the Friedman paper.
References
1 Friedman AS. A Difference-in-Differences Analysis of Youth Smoking and a Ban on Sales of Flavoured Tobacco Products in San Francisco, California. JAMA Pediatr 2021;175:863–5. doi:10.1001/jamapediatrics.2021.0922
2 Friedman AS. Further Considerations on the Association Between Flavoured Tobacco Legislation and High School Student Smoking Rates—Reply. JAMA Pediatr 2021;175:1291–2. doi:10.1001/jamapediatrics.2021.3293
3 Maa J, Gardiner P. Further Considerations on the Association Between Flavoured Tobacco Legislation and High School Student Smoking Rates. JAMA Pediatr 2021;175:1289–90. doi:10.1001/jamapediatrics.2021.3284
4 Liu J, Hartman L, Tan ASL, et al. In reply: Youth tobacco use before and after flavoured tobacco sales restrictions in Oakland, California and San Francisco, California. Tob Control Published Online First: 16 March 2022. doi:10.1136/tobaccocontrol-2021-057135
5 Gammon DG, Rogers T, Gaber J, et al. Implementation of a comprehensive flavoured tobacco product sales restriction and retail tobacco sales. Tob Control Published Online First: 4 June 2021. doi:10.1136/tobaccocontrol-2021-056494
6 Guydish JR, Straus ER, Le T, et al. Menthol cigarette use in substance use disorder treatment before and after implementation of a county-wide flavoured tobacco ban. Tob Control 2021;30:616–22. doi:10.1136/tobaccocontrol-2020-056000
7 Yang Y, Lindblom EN, Salloum RG, et al. The impact of a comprehensive tobacco product flavour ban in San Francisco among young adults. Addict Behav Rep 2020;11:100273. doi:10.1016/j.abrep.2020.100273
8 Holmes LM, Lempert LK, Ling PM. Flavoured Tobacco Sales Restrictions Reduce Tobacco Product Availability and Retailer Advertising. Int J Environ Res Public Health 2022;19:3455. doi:10.3390/ijerph19063455
9 Mantey DS, Kelder SH. Further Considerations on the Association Between Flavoured Tobacco Legislation and High School Student Smoking Rates. JAMA Pediatr 2021;175:1290. doi:10.1001/jamapediatrics.2021.3287
10 Leas EC. Further Considerations on the Association Between Flavoured Tobacco Legislation and High School Student Smoking Rates. JAMA Pediatr 2021;175:1290–1. doi:10.1001/jamapediatrics.2021.3290
11 Zhu S-H, Braden K, Zhuang Y-L, et al. Results of the Statewide 2019-2020 California Student Tobacco Survey. https://www.cdph.ca.gov/Programs/CCDPHP/DCDIC/CTCB/CDPH%20Document%20Lib...
12 Zhu S-H, Zhuang Y-L, Braden K, et al. Results of the Statewide 2017-2018 California Student Tobacco Survey. https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&ved=2ahUK...
13 Monitoring the Future (MTF) Public-Use Cross-Sectional Datasets. https://www.icpsr.umich.edu/web/NAHDAP/series/35 (accessed 1 Jul 2022).
14 Chan GCK, Stjepanović D, Lim C, et al. Gateway or common liability? A systematic review and meta-analysis of studies of adolescent e-cigarette use and future smoking initiation. Addiction 2021;116:743–56. doi:10.1111/add.15246
15 Soneji S, Barrington-Trimis JL, Wills TA, et al. Association Between Initial Use of e-Cigarettes and Subsequent Cigarette Smoking Among Adolescents and Young Adults: A Systematic Review and Meta-analysis. JAMA Pediatr 2017;171:788–97. doi:10.1001/jamapediatrics.2017.1488
16 Centers for Disease Control and Prevention. Youth Online: High School YRBS - Oakland, CA 2017 and 2019 Results Current Electronic Vapor Product Use. https://nccd.cdc.gov/Youthonline/App/Results.aspx?TT=A&OUT=0&SID=HS&QID=... (accessed 1 Jul 2022).
17 Centers for Disease Control and Prevention. Youth Online: High School YRBS - Oakland, CA 2017 and 2019 Results Current Cigarette Smoking. https://nccd.cdc.gov/Youthonline/App/Results.aspx?TT=A&OUT=0&SID=HS&QID=... (accessed 1 Jul 2022).
18 Centers for Disease Control and Prevention. Trends in the Prevalence of Tobacco Use National YRBS: 1991—2019. 2021.https://www.cdc.gov/healthyyouth/data/yrbs/factsheets/2019_tobacco_trend... (accessed 20 Jun 2022).
19 Centers for Disease Control and Prevention. Data Request and Contact Form- YRBSS. 2021.https://www.cdc.gov/healthyyouth/data/yrbs/contact.htm (accessed 1 Jul 2022).
NOT PEER REVIEWED
On March 17th, 2021, Tobacco Control published a paper online revealing that the 2019 wave of the Youth Risk Behavior Surveillance System (YRBSS) in San Francisco was fielded in the fall of 2018, as opposed to spring of 2019 as is typical for that survey. [1] On March 21st 2022, I received confirmation from San Francisco’s YRBSS site coordinator that the 2019 wave was fielded from November 5th, 2018 to December 14th, 2018. I appreciate Liu and colleagues bringing this to light. However, their claim that this information invalidates the findings from my 2021 JAMA Pediatrics paper [2] —linking San Francisco’s ban on sales of flavored tobacco and nicotine products to increases in youth cigarette smoking—is both methodologically and historically inaccurate: it overlooks both the assumptions required for difference-in-differences research designs and the full timeline of San Francisco’s flavor ban implementation.
In its simplest form, a difference-in-difference (DD) analysis of a particular policy compares outcomes in jurisdictions that did vs. did not adopt the policy, before vs. after that policy officially went into effect (See Figure at https://figshare.com/articles/figure/Figure_1_BasicDDExplanation_pdf/203...). If time-trends in the adopting and non-adopting jurisdictions’ outcomes were parallel in the pre-policy period, the non-adopters’ trends are c...
NOT PEER REVIEWED
On March 17th, 2021, Tobacco Control published a paper online revealing that the 2019 wave of the Youth Risk Behavior Surveillance System (YRBSS) in San Francisco was fielded in the fall of 2018, as opposed to spring of 2019 as is typical for that survey. [1] On March 21st 2022, I received confirmation from San Francisco’s YRBSS site coordinator that the 2019 wave was fielded from November 5th, 2018 to December 14th, 2018. I appreciate Liu and colleagues bringing this to light. However, their claim that this information invalidates the findings from my 2021 JAMA Pediatrics paper [2] —linking San Francisco’s ban on sales of flavored tobacco and nicotine products to increases in youth cigarette smoking—is both methodologically and historically inaccurate: it overlooks both the assumptions required for difference-in-differences research designs and the full timeline of San Francisco’s flavor ban implementation.
In its simplest form, a difference-in-difference (DD) analysis of a particular policy compares outcomes in jurisdictions that did vs. did not adopt the policy, before vs. after that policy officially went into effect (See Figure at https://figshare.com/articles/figure/Figure_1_BasicDDExplanation_pdf/203...). If time-trends in the adopting and non-adopting jurisdictions’ outcomes were parallel in the pre-policy period, the non-adopters’ trends are considered to be reasonable counterfactuals for the adopters’ trends. The corresponding multivariable regression explicitly controls for other policy changes that may affect the outcome, common time trends, and time-invariant differences between jurisdictions (i.e., absorbing ⍺ in Figure). In that context, changes in the adopting jurisdictions’ trends relative to non-adopters— β-⍺ in Figure 1 — can be attributed to the policy change. Such analyses use the policy’s official effective date as the pre- vs post-policy cut point to avoid confounding from endogenous delays in a policy’s implementation (e.g., as retailer or consumer behavior can contribute to implementation delays). In other words, a DD analysis based on realized enforcement dates risks introducing bias. Thus, official/legislated effective dates are used to ensure that resulting estimates capture unconfounded responses to the policy change.
While DD estimates are valid even when the official effective date precedes full implementation, claims about their generalizability may need to be constrained. In the case of San Francisco’s flavor ban, the implementation history suggests that the effects I estimated should be interpreted as responses to the partially implemented policy, as both the policy timeline and empirical data show responses to the policy in late-2018. Specifically, voters approved San Francisco’s ban on sales of flavored tobacco products via referendum on June 5th, 2018. While the policy’s legal effective date was July 21, 2018, the San Francisco Department of Public Health (SFDPH) announced that retailer violation penalties would not be enforced until January 1, 2019, so retailers could liquidate their existing stocks of flavored products. In the interim, SFDPH conducted retailer education and outreach starting in September 2018, and began compliance inspections on December 3rd, 2018. Retailers still selling flavored products at that point were informed that the flavor ban was in effect and they would face suspension of their tobacco sales permit if they continued to offer flavored products; and they were issued a Compliance Notification Letter with instructions to text a particular number to confirm compliance. Accordingly, San Francisco’s flavored tobacco product sales fell markedly in the second half of 2018: weekly averages for November and December 2018 were both well below those for the four weeks preceding July 21, 2018, a pattern not evident in comparison districts. [3] Retailer compliance was measured at 17% in December 2018 which, while low, still evinces a retailer response to the law before 2019. [4] Prior work showing that consumers respond to anticipated tobacco policy changes, not merely those already in effect, offers further ways San Francisco’s law could have affected consumer behavior during this period. [5]
Indeed, evidence on retailer behavior shows that enforcement per se was not necessary to induce retailer compliance. Specifically, despite SFDPH’s plan to begin enforcing retailer penalties in January 2019, the flavor policy’s Rules and Regulations were not finalized until August 16, 2019, meaning that non-compliant retailers did not face suspension of their tobacco sales permits in the first half of 2019 (Jennifer Callewaert, Principal Environmental Health Inspector at SFDPH, personal communication, 5/19/2022). Yet Vyas et al. (2021) document retailer compliance rates of 77%, 85%, and 100% in January, February, and March of 2019, respectively. [4] Thus, while expected penalties may have driven compliance during this period, enforcement per se could not have.
Liu et al.’s (2022) article cannot refute these mechanisms: beyond its failure to present any statistically significant evidence, the authors overlook the fact that youth cigarette smoking also declined in California districts without a flavor restriction during this period: from 2017 to 2019, YRBSS smoking rates dropped from 4.2% to 3.2% in San Diego, and 2.7% to 2.3% in Los Angeles. Thus, common time trends could explain Oakland’s nonsignificant trend, as opposed to its flavor policy. Perhaps more importantly, Oakland’s law was substantively different from San Francisco’s: the former allowed retailer exemptions and thus may have created different incentives for illicit suppliers—e.g., if a lack of legal sources for adults makes illicit sales of menthol cigarettes more profitable—yielding different effects on underage access. In this context, even if perfect estimates of the Oakland and San Francisco policies’ effects differed, one would not constitute evidence against the other because the policies themselves are different.
It is worth exploring conceptually why youth cigarette smoking might increase in response to a comprehensive flavor ban. Informal market responses to this policy offer one potential mechanism: if flavor bans make flavored products more profitable for illicit sellers, they could increase underage access to flavored combustible products (e.g., if illicit sellers stock up on menthol loosies, combustible menthol products may have actually become more accessible post-ban for youth who rely on unlicensed sellers). Alternatively, youth who preferred flavored products might turn to flavor accessories primarily designed for use with combustible products (e.g., flavor cards, crush balls), making smoking more attractive relative to vaping once flavored vapes were not offered by licensed retailers (particularly if the 2019 outbreak of vaping-associated lung injuries reduced people’s willingness to buy vaping products from informal sellers).
Youth substitution from exclusive cigar use towards cigarettes might explain a portion but not all of the results: as the majority of youth cigar users already smoke, the effect size I estimated is too large to be fully explained by youth who previously smoked cigars. While substitution could not be assessed directly (as San Francisco’s YRBSS data did not cover cigar use in 2015-2019), over 70% of San Francisco minors responding to the 2013 YRBSS who reported past 30 day cigar use already smoked cigarettes. Rescaling these numbers based on 2013 to 2017 reductions in cigar use observed in other California districts suggests that about 0.7% of San Francisco youths smoked cigars but not cigarettes in 2017. If all of them switched to cigarettes in response to the flavor ban, it would account for less than 15% of my effect estimate. (I derived these estimates based on YRBSS data).
Finally, it is possible that San Francisco youth who took up smoking in late 2018 were already addicted to nicotine, and simply switched to cigarettes as the most accessible substitute once flavored ENDS were no longer on the market. In that case, flavor restrictions’ long-run effects might differ from the short run if the lack of flavored ENDS reduces youth nicotine uptake. This is an important possibility that calls for further study.
My paper certainly is not the final say on flavor restrictions’ effects. As the original article noted, its findings may not generalize in the long run, to other jurisdictions, or to heterogeneous flavor restrictions. It provides one piece of evidence on how minors’ cigarette smoking changed under one partially implemented flavor policy in a distinctive urban center. We need more research on longer run outcomes across many different jurisdictions’ policies, considering both youth and adult behavior as well as effects on the illicit market, to fully understand flavor restrictions’ implications for public health.
Funding Statement: This research was supported by the National Institute on Drug Abuse of the National Institutes of Health (grant 3U54DA036151-08S2) and the US Food and Drug Administration Center for Tobacco Products. The funders had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
References
[1] Liu J, Hartman L, Tan ASL, et al. Youth tobacco use before and after flavoured tobacco sales restrictions in Oakland, California and San Francisco, California. Tob Control 2022 [Published Online First]. 17 March 2022 [cited 2022 June 16] http://dx.doi.org/10.1136/tobaccocontrol-2021-057135.
[2] Friedman AS. A difference-in-differences analysis of youth smoking and a ban on sales of flavored tobacco products in San Francisco, California. JAMA Pediatr 2021;175(8):863-865.
[3] Gammon DG, Rogers T, Gaber J, et al. Implementation of a comprehensive flavoured tobacco product sales restriction and retail tobacco sales. Tob Control, 2021 [Published Online First]. 4 June 2021 [cited 2022 June 16] http://dx.doi.org/10.1136/tobaccocontrol-2021-056494.
[4] Vyas P, Ling P, Gordon B, et al. Compliance with San Francisco’s flavoured tobacco sales prohibition. Tob Control 2021;30:227-230.
[5] Gruber J, Köszegi B. Is addiction "rational"? Theory and evidence. Q J Econ 2001;116(4): 1261-1303.
NOT PEER REVIEWED
If PMI attempted to profit from the “EVALI” scaremongering they could only do so because of the blatantly dishonest reporting of that issue by federal authorities, activist academics, tobacco control organisations and the media who quote them with question. It was obvious as early as August 2019 that the lung injuries were caused by black market THC cartridges cut with vitamin E acetate and not nicotine containing e-cigarettes and the CDC eventually came to the same conclusion. Yet activists in positions of authority continue to link the injuries with nicotine vaping, thus providing a fertile ground of misinformation in which such marketing campaigns can flourish.
A brief review of this ‘Industry Watch’ article alleging heated tobacco product advertising through an earned media approach highlights significant methodological errors that are serious enough to invalidate the article’s conclusions, including its title. The authors allege that Philip Morris International (PMI) used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to promote IQOS in September 2019 and the weeks that followed. Using the authors’ own tool (TobaccoWatcher.org), we replicated their search strategy and revealed several fundamental and concerning errors in the authors’ analysis.
They report a rise in news stories mentioning IQOS on and after 25th September 2019, and falsely attribute this rise to an article published on our website on 24th September 2019, which they also falsely describe as a “press release”, despite it never being published through a press release distribution service. Our analysis shows that the authors failed to consider several confounding and unrelated events that caused the rise in news coverage of both IQOS and EVALI during the time period in question and which can be found by replicating the authors’ search strategy in TobaccoWatcher.org.
For example, on 25th September 2019, Philip Morris International (PMI) issued a single press release via Business Wire (1) entitled “Philip Morris International Inc. and Altria Group, Inc. End Merger Discussions” (PMI/Altria Annou...
A brief review of this ‘Industry Watch’ article alleging heated tobacco product advertising through an earned media approach highlights significant methodological errors that are serious enough to invalidate the article’s conclusions, including its title. The authors allege that Philip Morris International (PMI) used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to promote IQOS in September 2019 and the weeks that followed. Using the authors’ own tool (TobaccoWatcher.org), we replicated their search strategy and revealed several fundamental and concerning errors in the authors’ analysis.
They report a rise in news stories mentioning IQOS on and after 25th September 2019, and falsely attribute this rise to an article published on our website on 24th September 2019, which they also falsely describe as a “press release”, despite it never being published through a press release distribution service. Our analysis shows that the authors failed to consider several confounding and unrelated events that caused the rise in news coverage of both IQOS and EVALI during the time period in question and which can be found by replicating the authors’ search strategy in TobaccoWatcher.org.
For example, on 25th September 2019, Philip Morris International (PMI) issued a single press release via Business Wire (1) entitled “Philip Morris International Inc. and Altria Group, Inc. End Merger Discussions” (PMI/Altria Announcement). The release noted that “the companies have agreed to focus on launching IQOS in the U.S.” PMI is listed on the New York Stock Exchange and is therefore subject to the rules of the NYSE and the Securities and Exchange Commission, including requirements for the disclosure of material events such as the PMI/Altria Announcement. We estimate that during a date range similar to that used by the authors, of the total number of stories retrieved using the authors’ search terms, >85% were reporting on the above noted PMI/Altria Announcement. The authors failed to mention this and other confounders in their discussion.
The authors state that “the original PMI press release (sic) was republished (including editing or revisions to make the republication consistent with the publisher’s format) in 14 additional news outlets.” We performed a full text review of the 14 articles mentioned and of those that were still accessible, none mentioned IQOS. The authors’ conclusion that “PMI capitalised on EVALI by using an earned news media strategy to market their IQOS brand of heated tobacco for free across scores of news outlets that reached millions of readers. This marks the first known case where a tobacco company used the EVALI outbreak to promote tobacco products” is therefore completely unsupported by their own data.
These facts, and other errors detected in this article, were communicated to the authors on 23rd April 2021. In summary, we believe that the article is unreliable, and that urgent substantial revision or retraction is warranted to prevent other studies citing the currently published inaccuracies.
On behalf of my co-authors, we thank Dr. Mishra for taking the time to comment on our paper examining reporting of trials registered by Juul Labs Inc.
We do not doubt that Juul has submitted the results from all of these studies to the FDA as part of their Premarket Tobacco Product Application (PMTA). Unfortunately, this step does not ensure the full results of these trials will be made available to the public, clinicians, and other key stakeholders.
We also acknowledge and appreciate that since conducting our analysis in August 2020, two additional studies, of the five we examined, have appeared in the peer reviewed literature (published in December 2020 and January 2021)[1,2] and additional results may become available through other methods. We understand that these publications may contain additional outcome reporting compared to the posters we assessed. We welcome any and all additional results disclosures from Juul Labs. That said, we believe the risk of outcome reporting bias remains and is, unfortunately, not addressed in Juul’s comment despite being a major facet of our paper. The one publication that was available as of our analysis date excluded 4 of the 19 prespecified secondary outcomes without comment and another 5 outcomes had clear issues in their reporting compared to the registered outcomes. While we have not conducted a detailed assessment of the two new publications, it is apparent that journal publication does no...
On behalf of my co-authors, we thank Dr. Mishra for taking the time to comment on our paper examining reporting of trials registered by Juul Labs Inc.
We do not doubt that Juul has submitted the results from all of these studies to the FDA as part of their Premarket Tobacco Product Application (PMTA). Unfortunately, this step does not ensure the full results of these trials will be made available to the public, clinicians, and other key stakeholders.
We also acknowledge and appreciate that since conducting our analysis in August 2020, two additional studies, of the five we examined, have appeared in the peer reviewed literature (published in December 2020 and January 2021)[1,2] and additional results may become available through other methods. We understand that these publications may contain additional outcome reporting compared to the posters we assessed. We welcome any and all additional results disclosures from Juul Labs. That said, we believe the risk of outcome reporting bias remains and is, unfortunately, not addressed in Juul’s comment despite being a major facet of our paper. The one publication that was available as of our analysis date excluded 4 of the 19 prespecified secondary outcomes without comment and another 5 outcomes had clear issues in their reporting compared to the registered outcomes. While we have not conducted a detailed assessment of the two new publications, it is apparent that journal publication does not necessarily guarantee the full reporting of all specified outcomes. This unfortunately holds true throughout the reporting of clinical research. [3, 4]
Additionally, we showed that for the conference posters identified by our searches there was a clear risk of unreported outcomes for these short-format publications. Potentially making some studies only available as posters or abstracts risks further outcome reporting issues or delays.
In Footnote 9 of the comment, Dr. Mishra states that “Section 801 of the Food and Drug Administration Amendments Act of 2007 (FDAAA) and implementing regulations issued by the National Institutes of Health (NIH) do not require registration of clinical trials involving tobacco products”
The Final Rule is entirely silent on the specific coverage status of tobacco products. Searching the Final Rule document for the word “Tobacco” reveals 0 results.[5] Another Final Rule issued well after the Sottera decision, 21 CFR 201, 801, and 1100, and an FDA guidance document both make clear that under certain circumstances tobacco products can be considered covered under the FDAAA.[6, 7]
To quote the FDA Guidance:
“If an ENDS product is marketed for tobacco cessation or for any other therapeutic purpose, the product is a drug or device, rather than a tobacco product, under the authorities of FDA’s Center for Drug Evaluation and Research or Center for Devices and Radiological Health, and appropriate approval must be sought to market a product as a drug or device.” (Page 9)
And 21 CFR 201, 801, and 1100:
“Whether the product, as used in the study, is subject to regulation as a drug depends on whether the product is being investigated for any of the purposes described in § 1100.5(a) or (b) of this rule. To determine if a product made or derived from tobacco is being investigated for one of these purposes, FDA generally would review the protocol for the study, including the proposed methods and measures. In the Agency’s experience, the proposed methods and measures for a study can provide insight into the purposes for which a product is being investigated.” (Page 2213)
We also note that the FDAAA Final Rule requires trials of drugs and devices to report regardless of whether or not they plan to seek any approvals with the FDA [5]:
“The final rule maintains the proposal to require the submission of results information for applicable clinical trials of unapproved, unlicensed, or uncleared products, regardless of whether FDA approval, licensure, or clearance is or will be sought or obtained.” (Page 64995)
We discuss the relevant legal frameworks, and their unfortunate ambiguity, in more depth in the manuscript.
Our determination of potential FDAAA coverage was based on public registration information directly provided by Juul and matched to official published criteria for determining FDAAA coverage. As stated in their comment, Juul Labs have now updated many of their registrations to no longer match these criteria. However the registered trials we assessed contained outcomes related to the uptake and pharmacokinetics of nicotine in the body, a key feature of addiction, and measures directly relevant to smoking cessation like product use, dependency, urge to smoke, and aspects of withdrawal. It is, however, ultimately up to the FDA to make official determinations about which trials are covered under the FDAAA based on their assessment of the purpose and outcomes of a given study.
Regardless of the exact regulatory status of these trials, a matter we hope the FDA will offer further clarity on, we strongly encourage Juul Labs to consider amending their existing registrations with the results of these trials. ClinicalTrials.gov offers a rapid, public, open source repository for trial results that enhances their availability, transparency, and discoverability and should not preclude subsequent peer reviewed publications.[8] It is clear from their response that Juul Labs understands the importance of trial registration and reporting to the public and scientific community, and we welcome this recognition. ClinicalTrials.gov allows for full results to be reported, without space limitations or delays associated with journal publication, and with a clear public link to all prespecified outcomes and other registered details. While the FDAAA compels this type of reporting, there is no limitation on voluntary submission of results if trials are not covered under the law. We strongly believe it is in the public interest that all outcomes for all trials that form PMTA applications should be fully and publicly reported in a timely manner. Currently this does not appear to be the case even if it may have improved since our evaluation.
Whether compelled to report by legal requirements or not, we hope Juul will consider setting a strong precedent for transparency in tobacco-industry e-cigarette research by adhering to global ethical standards [9] and reporting their full trial results for these, and future, trials to ClinicalTrials.gov and assuage any doubts about selective reporting of results or outcomes.
I write on behalf of Juul Labs in response to “E-cigarette manufacturers’ compliance with clinical trial reporting expectations: a case series of registered trials by Juul Labs.” We appreciate the public health community’s interest in understanding the Company’s research. However, we want to clarify the article’s reporting on Juul Labs’ research program and publications to date, as well as our legal obligations to report the results of clinical investigations.
First, our studies were conducted in anticipation of our submission of a Premarket Tobacco Product Application (PMTA) to the U.S. Food and Drug Administration (FDA). As such, raw data, results, and statistical code underlying our analyses of all ten registered Juul Labs studies referenced in the Tobacco Control paper have been submitted to FDA (1)
In addition, we have been working on sharing our data in publications and posters. In fact, by the end of this month, we will have published or presented data on nine of the ten studies, in addition to having provided the clinical results to FDA. Of our ten registered studies, we have published the results of four (2, 3, 4, 5) submitted the results of an additional two studies for peer review, presented the results of two as posters (6,7) and have the results of one accepted for presentation this summer. We appreciate that the authors of the Tobacco Control paper may have drafted their article months ago, but note that there...
I write on behalf of Juul Labs in response to “E-cigarette manufacturers’ compliance with clinical trial reporting expectations: a case series of registered trials by Juul Labs.” We appreciate the public health community’s interest in understanding the Company’s research. However, we want to clarify the article’s reporting on Juul Labs’ research program and publications to date, as well as our legal obligations to report the results of clinical investigations.
First, our studies were conducted in anticipation of our submission of a Premarket Tobacco Product Application (PMTA) to the U.S. Food and Drug Administration (FDA). As such, raw data, results, and statistical code underlying our analyses of all ten registered Juul Labs studies referenced in the Tobacco Control paper have been submitted to FDA (1)
In addition, we have been working on sharing our data in publications and posters. In fact, by the end of this month, we will have published or presented data on nine of the ten studies, in addition to having provided the clinical results to FDA. Of our ten registered studies, we have published the results of four (2, 3, 4, 5) submitted the results of an additional two studies for peer review, presented the results of two as posters (6,7) and have the results of one accepted for presentation this summer. We appreciate that the authors of the Tobacco Control paper may have drafted their article months ago, but note that there are outdated references to publication or presentation of the five studies included by the authors in their analysis. While the article asserts that just one of the five studies that was incorrectly registered as an “applicable device clinical trial” has been published, as of this writing, results from three of the five studies have been published. (8) Across our research program, we have published a total of 17 articles (all publicly available on our science website www.juullabsscience.com) and are working on additional publications.
Finally, although not obligated to do so (9), Juul Labs voluntarily submitted registration information for ten studies to the ClinicalTrials.gov registry. As voluntary submissions, all of the studies should have been registered as a “voluntary other trial.” Due to an administrative error at Juul Labs, six of those studies were instead registered as an “applicable device clinical trial.” These errors are in the process of being corrected and, in fact, four of the studies have been formally updated on ClinicalTrials.gov to reflect this change. (10, 11, 12, 13)
We look forward to continuing to share results from our science and research program with the scientific and public health community as we work to advance harm reduction for adults who smoke, and to support the scientific basis for the category.
Sincerely, Snigdha Mishra, SVP Regulatory Science
A review of this study has been published by the target of its criticism, the tobacco company Philip Morris International (PMI), via the post-publication review server Qeios [1]
The main finding of this study, and the allegation raised in its title, is that PMI cynically used an outbreak of lung injuries in the United States (initially but incorrectly attributed to nicotine vaping) to promote its heated tobacco product, iQOS. Heated tobacco products are one alternative to vaping for those looking for a safer alternative to smoking. On 24th September 2019, PMI published an information notice about its products in response to the lung injury outbreak. The authors assert that PMI was trying to gain commercial publicity from a health crisis: a serious allegation. But the allegation appears to be based on a major error by the authors.
The study used a "fully automated media analysis engine" to count stories that mention iQOS around that time, showing that there were considerably more than usual. On this basis, the authors concluded that PMI's unethical promotional gambit had worked. However, the day after PMI allegedly disreputably sought publicity for iQOS, the company also issued a press release disclosing that merger negotiations with the American tobacco company, Altria, had ceased. PMI and Altria have a joint marketing agreement for iQOS in the United States. The end of merger talks would be big news in the business pre...
A review of this study has been published by the target of its criticism, the tobacco company Philip Morris International (PMI), via the post-publication review server Qeios [1]
The main finding of this study, and the allegation raised in its title, is that PMI cynically used an outbreak of lung injuries in the United States (initially but incorrectly attributed to nicotine vaping) to promote its heated tobacco product, iQOS. Heated tobacco products are one alternative to vaping for those looking for a safer alternative to smoking. On 24th September 2019, PMI published an information notice about its products in response to the lung injury outbreak. The authors assert that PMI was trying to gain commercial publicity from a health crisis: a serious allegation. But the allegation appears to be based on a major error by the authors.
The study used a "fully automated media analysis engine" to count stories that mention iQOS around that time, showing that there were considerably more than usual. On this basis, the authors concluded that PMI's unethical promotional gambit had worked. However, the day after PMI allegedly disreputably sought publicity for iQOS, the company also issued a press release disclosing that merger negotiations with the American tobacco company, Altria, had ceased. PMI and Altria have a joint marketing agreement for iQOS in the United States. The end of merger talks would be big news in the business press and would likely include a discussion of consequences for iQOS.
So, an analysis that tries to measure media interest in PMI and iQOS for cynical publicity purposes at that time would be heavily confounded by other unrelated but newsworthy developments concerning the same company and product. A check of the press releases by the company should have raised the alarm. PMI also states that it did not seek any publicity for its safety notice about iQOS, which PMI says the authors:
"falsely describe as a “press release”, despite it never being published through a press release distribution service"
Now that it has been pointed out, it is quite clear that a serious error has been made here that undermines the foundations of the study and renders the allegation made in the title baseless. The simple way to deal with this is to acknowledge the error, retract the article, apologise, and resolve to be more careful in future. That should take a couple of hours, not several months and there would be no shame in it. That this has not happened is troubling.
[1] Moira Gilchrist (PMI) Review of: Philip Morris International used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to market IQOS heated tobacco. Qeios. [https://www.qeios.com/read/NLZDBR}
We thank Dr. Moira Gilchrist (1) for her careful attention to our work (2). Gilchrist argues our principal findings were erroneous and any change in news coverage of IQOS and the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak were confounded by other Philip Morris International (PMI) media materials and not those specifically discussing EVALI and IQOS (3) which we attributed our findings to. However, a deeper inspection of this argument suggests our original findings and conclusions remain plausible.
Tobacco Watcher is a dynamic resource with continuous data collection and processing. Thus, the results of analyses on the platform can vary over time. On June 10, 2021 we replicated our analysis. After correcting an error that the PMI’s materials on EVALI and IQOS (3) was initially published on 24 September 2021 (not 25 September) the principal finding is unchanged. News coverage mentioning both “IQOS” and EVALI (i.e., including the terms ‘vaping’ and ‘illness’) reached an all-time high immediately after PMI published materials about EVALI and IQOS on their website. Thirty days prior to PMI posting this material (August 25th through September 23rd) 2.0 news stories per day matched our search compared to 12.8 for the 30 days after their publication (September 24th through October 23rd), with 384 news reports matching our keyword search for the latter period. Our original assertion that there were 14 duplicate articl...
We thank Dr. Moira Gilchrist (1) for her careful attention to our work (2). Gilchrist argues our principal findings were erroneous and any change in news coverage of IQOS and the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak were confounded by other Philip Morris International (PMI) media materials and not those specifically discussing EVALI and IQOS (3) which we attributed our findings to. However, a deeper inspection of this argument suggests our original findings and conclusions remain plausible.
Tobacco Watcher is a dynamic resource with continuous data collection and processing. Thus, the results of analyses on the platform can vary over time. On June 10, 2021 we replicated our analysis. After correcting an error that the PMI’s materials on EVALI and IQOS (3) was initially published on 24 September 2021 (not 25 September) the principal finding is unchanged. News coverage mentioning both “IQOS” and EVALI (i.e., including the terms ‘vaping’ and ‘illness’) reached an all-time high immediately after PMI published materials about EVALI and IQOS on their website. Thirty days prior to PMI posting this material (August 25th through September 23rd) 2.0 news stories per day matched our search compared to 12.8 for the 30 days after their publication (September 24th through October 23rd), with 384 news reports matching our keyword search for the latter period. Our original assertion that there were 14 duplicate articles is not supported by our replication analysis.
We recognize that for any observational data analysis there are alternative explanations. However, Gilchrist’s alternative explanations are debatable. First, she argues the materials published by PMI on EVALI and IQOS (3) were not a “press release” and therefore could not have engendered a change in media coverage. We now refer to the materials published by PMI on EVALI and IQOS as a “public statement” published on PMI’s website. Yet, PMI has not provided any verifiable disclosure about how they circulated these materials beyond their publication and availability on internet search engines and social media, a concern we raised in our study (2). Regardless, PMI’s public statement was potentially consumed and amplified by other media makers. Second, she assumes the increase in news coverage was due to a PMI publication entitled “Philip Morris International Inc. and Altria Group, Inc. End Merger Discussions” (4) which does not mention EVALI. While a sizable percentage of the news articles we studied mention the merger, their inclusion of IQOS and EVALI suggests the possibility of additional source material for these stories. It is plausible that information from the published materials we studied tying IQOS and EVALI and the larger merger story were discussed in tandem.
There were factual errors in our report that we missed during the copy edit of our piece: the date of PMI’s materials being published was misstated (25 versus 24 September), the date of another reference was misstated, and references were misordered. Additionally, the date we submitted our piece was misstated due to a production system issue created by a delay in submitting the revised version. Last, in light of PMI stating that the 25 September publication was not a press release, the article has been amended accordingly. We have provided Tobacco Control with an errata to correct these errors. However, the principal conclusion of our work was unaffected by any error.
While we can measure the potential spillover effect of published claims, we cannot know PMI’s intent. It is paramount that the earned media strategies of the tobacco industry come under closer inspection. The tobacco industry has used varied tactics to sow misimpressions that favor them and their products, a fact well documented in both the scientific literature and courts of law (5); earned media campaigns may be yet another.
References
1. Gilchrist M. Study alleging Philip Morris International used the EVALI outbreak to market IQOS requires substantial methodological revision and further peer review, or retraction. Tob Control Rapid Response.
2. Ayers JW, Leas EC, Dredze M, et al. Tob Control. Epub ahead of print: doi:10.1136/ tobaccocontrol-2021-056661.
3. Philip Morris International. Lung illnesses associated with use of vaping products in the US. Available: https://www.pmi.com/media-center/news/lung-illnesses-associated-with-use... [Accessed 10 Mar 2021].
4. Philip Morris International. Philip Morris International Inc. and Altria Group, Inc. End Merger Discussions. https://philipmorrisinternational.gcs-web.com/static-files/78a6afb3-107d... Accessed 11 June 2021.
5. Ong EK, Glantz SA. Constructing "sound science" and "good epidemiology": tobacco, lawyers, and public relations firms. Am J Public Health. 2001;91.
NOT PEER REVIEWED
The authors’ response published on 14 July 2021 is far from satisfactory and implausibly asserts that “Our original findings and conclusions remain plausible” [1]
The original study [2] uses a media analysis to make a claim that a statement made by the tobacco company Philip Morris International (PMI) about an outbreak of lung disease in the US [3] was a marketing ploy for its heated tobacco product, iQOS. At the time, the lung injury outbreak was falsely attributed by many to nicotine vaping. Heated tobacco products are an alternative to nicotine vaping for smokers looking for a low-risk alternative to smoking.
I will now list some of the problems with this claim.
1. The research findings do not support the headline claim
The study title contains a strong and unqualified assertion of cynical opportunism on the part of the company. The new formulation that findings "remain plausible" does not justify the confidence in the assertion made in the title. "Plausible" is a reasonable basis for choosing a hypothesis to investigate, but a far from sufficient basis for drawing an aggressive conclusion. The authors do not seem to dispute the technical or factual accuracy of the statement about iQOS and EVALI made by PMI. Their allegation is about malign motives and, as such, it should be a cause for caution and a high standard of evidence.
2. No specific articles were provided to substantiate the...
NOT PEER REVIEWED
The authors’ response published on 14 July 2021 is far from satisfactory and implausibly asserts that “Our original findings and conclusions remain plausible” [1]
The original study [2] uses a media analysis to make a claim that a statement made by the tobacco company Philip Morris International (PMI) about an outbreak of lung disease in the US [3] was a marketing ploy for its heated tobacco product, iQOS. At the time, the lung injury outbreak was falsely attributed by many to nicotine vaping. Heated tobacco products are an alternative to nicotine vaping for smokers looking for a low-risk alternative to smoking.
I will now list some of the problems with this claim.
1. The research findings do not support the headline claim
The study title contains a strong and unqualified assertion of cynical opportunism on the part of the company. The new formulation that findings "remain plausible" does not justify the confidence in the assertion made in the title. "Plausible" is a reasonable basis for choosing a hypothesis to investigate, but a far from sufficient basis for drawing an aggressive conclusion. The authors do not seem to dispute the technical or factual accuracy of the statement about iQOS and EVALI made by PMI. Their allegation is about malign motives and, as such, it should be a cause for caution and a high standard of evidence.
2. No specific articles were provided to substantiate the claim
The headline finding is not, however, even remotely plausible. The company has made it clear that it made its iQOS-EVALI statement available passively for information purposes. If it had generated even one story among 384 the authors say that matched their keyword search in the 30 days after publication, the authors could have produced it. If they had a good case and wanted to avoid a retraction, they could have looked through these results (or to save time, articles published in the two weeks after PMI's statement) and shown at least some articles clearly derive from the statement. Though searching only with Google, I have been unable to identify a single article. That doesn't mean there aren't any, but it's not me making the allegations. The authors are making the allegations and the burden of evidence is with them. They should now respond with examples of articles that support their critique and publish the list of 384 articles captured in their search as a supplementary file that is open to anyone to examine
3. Confounding by other news is a fatal problem for this study
The problem of confounding by other news related to the company, product and sector at around the same time is overwhelming. PMI published its iQOS-EVALI statement on 24 September 2019 [3]. This period coincided with; the end of merger talks between Altria and PMI on 25 September [4]; also on the same day, Juul announced the departure of its CEO and new corporate practices (Juul is part-owned by Altria) [5]; a new iQOs product was launched by PMI on 26 September [6]; there was an ongoing controversy with high news value developments on a lung injury outbreak in the United States, which was at the time often falsely attributed to nicotine vaping. For example, a significant CDC press release on 27 September [7].
4. Many news media will have covered several issues at once
The timing in question was a crisis period for products and companies making low-risk alternatives to cigarettes. It will have generated many news pieces that would have covered specific stories but also rounded up the other news. For example, Reuters reported both the PMI-Altria merger news and Juul restructuring in the same piece on 25 September 2019 with references to both lung injures and to iQOS, but without any reference to PMI's 24 September 2019 statement. [8]
5. The study method has no credible approach to confounding
Much of the response seems to be devoted to discussing sloppy though trivial errors in the original paper. This is a distraction from addressing the fundamental flaw, on which the response has relatively little to say. Without some sort of strategy to address confounding by other news and thereby to isolate from the signal (of PMI cynical opportunism) from the cacophonous noise of the news flow at the time, the authors have no basis for their claim. On top of this, the failure to produce a single article that appears to be derived in whole or even in part from the allegedly cynical communication suggests the authors have nothing.
6. Failure to put findings to the company
There is no sign here that the authors contacted the company or made any efforts to verify their story or check for alternative explanations. Why not? Why is it acceptable to avoid this most basic of journalistic practices simply because the authors are writing in an academic journal and about an organisation they do not like?
Just because the object of this research is the tobacco industry, it doesn't mean scientific standards or research and investigative ethics should be jettisoned. Precisely because the target of this research is a tobacco company, it makes a good test of scientific integrity and ‘white hat bias’. [9]
“White Hat Bias is bias leading to distortion of research-based information in the service of what may be perceived as righteous ends.”
[9]. The concept originates with: Cope MB, Allison DB. White Hat Bias: Examples of its Presence in Obesity Research and a Call for Renewed Commitment to Faithfulness in Research Reporting. Int J Obes (Lond) 2010;34(1):84. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2815336/
NOT PEER REVIEWED
Replication attempts are one of the self-correcting mechanisms of science, and we thank the Authors for their response to our concerns and their attempt to replicate aspects of their study [1]. Regrettably, they have failed to adequately address the central point raised in our letter of 23rd April 2021, namely that the title and conclusions of their original Article are patently invalid and have no basis in fact or evidence [2]. Instead of strengthening their argument in support of the Article’s findings and conclusions, the Authors’ response considerably weakens them. Strikingly, the Authors reveal several new and serious issues and yet maintain that their “principle finding is unchanged”.
Methodological Problems:
The Authors acknowledge that they were unable to replicate an important aspect of their original analysis, namely that a Philip Morris International (PMI) News Article [3] published on its website (falsely described as a “press release”) was “republished […] in 14 additional news outlets”. In their response, they note that “Our original assertion that there were 14 duplicate articles is not supported by our replication analysis”. This failure to replicate a key finding—in their own proprietary database, which several of them co-developed—is concerning. The Authors provide no explanation for the irregularity. Notably, on 20th April 2021, we were able to source these 14 articles in Tobacco Watcher since they were clearly mar...
NOT PEER REVIEWED
Replication attempts are one of the self-correcting mechanisms of science, and we thank the Authors for their response to our concerns and their attempt to replicate aspects of their study [1]. Regrettably, they have failed to adequately address the central point raised in our letter of 23rd April 2021, namely that the title and conclusions of their original Article are patently invalid and have no basis in fact or evidence [2]. Instead of strengthening their argument in support of the Article’s findings and conclusions, the Authors’ response considerably weakens them. Strikingly, the Authors reveal several new and serious issues and yet maintain that their “principle finding is unchanged”.
Methodological Problems:
The Authors acknowledge that they were unable to replicate an important aspect of their original analysis, namely that a Philip Morris International (PMI) News Article [3] published on its website (falsely described as a “press release”) was “republished […] in 14 additional news outlets”. In their response, they note that “Our original assertion that there were 14 duplicate articles is not supported by our replication analysis”. This failure to replicate a key finding—in their own proprietary database, which several of them co-developed—is concerning. The Authors provide no explanation for the irregularity. Notably, on 20th April 2021, we were able to source these 14 articles in Tobacco Watcher since they were clearly marked as “additional coverage” of the PMI News Article. Of the total, 5 of the “duplicate articles” were no longer accessible online and none of the remaining 9 mentioned IQOS at all. All of them were reporting on the EVALI outbreak and flavor ban proposals, a fact that we raised in our April letter to the Authors.
The Authors note that Tobacco Watcher is “a dynamic resource with continuous data collection and processing”. We could not find any information about the computational environment of the database, nor does there appear to be any public information about factors that can impact reproducibility and replicability. Specifically, we could not find any published information about how it collects, processes, and analyzes data, how it is version controlled and what data management plans are in place. However, a poster presentation by some of the Authors indicates that media articles are “automatically coded” using “natural language processing, trained on human-coded data” [4]. It is therefore difficult to understand why results relating to media articles from 2019—generated from an automatic coding process—could abruptly change over the course of two months in 2021. Of course, it is entirely plausible that the 14 articles had been falsely coded, and that this error was manually addressed at some point between April 20th and June 10th, 2021. The Authors should explain if this was the case, or whether changes to the underlying Tobacco Watcher algorithms caused the change—and if so, how. They should also provide an explanation of why they remain confident in the rest of their trend analyses, given the apparent instability of Tobacco Watcher over time.
Because of the Authors’ failure to replicate this important aspect of their study, there is no valid evidence to show any direct link between the PMI News Article and the increase in media coverage of IQOS and EVALI. The Authors’ invalid assumption that the PMI News Article was “republished” was presumably relied upon to support their title and conclusions and yet the evidence for this has now been shown to be nonexistent—not just by our analysis, but also by the Authors’ subsequent failure to replicate their initial finding.
Confounding Factors Ignored:
Given that the Authors have now shown that they do not have any evidence to support a direct, causal relationship between the PMI News Article and subsequent news coverage including both the terms IQOS and EVALI, potential confounders become central to the question of whether their title and conclusions are supported or not. Whilst they now concede that their evidence merely demonstrates that a causal relationship is “plausible”, they continue to ignore or dismiss other more likely explanations. Many confounding factors were laid out in our April letter—with sources beyond those emanating solely from PMI—and they provide important information about the news media environment that could credibly have caused a rise in articles mentioning IQOS and EVALI (including in particular the official press releases [5] by PMI and Altria on 25th September 2019—announcing that their proposed merger talks were over and that the companies would focus on launching IQOS in the U.S.— as well as a press release from the U.S. Centers for Disease Control related to EVALI). Generally, in epidemiology, sociology and other fields, there are at least three main criteria that should be met in order to establish a causal relationship: the cause occurs before the effect; the cause and the effect covary; and a lack of plausible alternative explanations [6]. By dismissing the most plausible explanations for the increase in media coverage, and failing to provide any credible alternative evidence to explain their hypothesis, the overall analysis and conclusion presented by the Authors fails to meet these criteria for causality.
Attempt to Shift the Burden of Proof:
The Authors make logical errors in defense of their Article. They state that “PMI has not provided any verifiable disclosure about how they circulated these materials [the PMI News Article] beyond their publication and availability on internet search engines and social media”. Any perceived lack of disclosure on the part of PMI does not render the Authors’ claims any more valid. To the contrary, the burden of proof is on them to provide evidence for their assertions. In the three months since we first wrote to the Authors to express our concerns, it would have been perfectly possible for them to: (a) list which news articles actually refer to the PMI News Article and/or use the phrases ‘IQOS’ AND ‘EVALI’ (rather than ‘vaping’ and ‘illness’, which are not identical search terms); and (b) rapidly and independently verify whether any of the journalists writing about both IQOS and EVALI had seen the PMI News Article by simply asking them.
Unsubstantiated, Yet Definitive Language:
The title “Philip Morris International used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to market IQOS heated tobacco” is a statement purporting to be factual. However, the Authors now state that their original findings and conclusions are merely “plausible” and admit “we cannot know PMI’s intent”. Plausibility is not a statement of fact or truth. Instead, it implies that an argument is not fully formed, appears specious, superficially fair or reasonable, but in fact could be either right or wrong. As such, the title and other conclusory allegations in the Article are—by the Author’s own admission—not supported. Having ignored or discounted evidence that makes the Author’s conclusions implausible, the final paragraph of their response makes clear that in the absence of evidence, their conclusion rests solely on the Author’s distrust of the industry.
Summary:
In summary, the conclusion that “Philip Morris International used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to market IQOS heated tobacco” can be disproven using the authors’ own tool, an analysis method that actually takes into account concomitant news events, the Authors’ acknowledgment that they failed to replicate an important aspect of their findings, and their concession that their title is merely “plausible”. The entire premise of the EVALI/IQOS Article is patently invalid and substantial revision or retraction of this paper remains urgently warranted.
NOT PEER REVIEWED
Show MoreThese arguments by Pesko and Friedman cannot undo the central flaw in the Friedman paper. We are surprised that Pesko and Friedman continue to argue that Friedman’s analysis of the YRBSS fall data as “after” data is valid despite the Friedman paper defining the exposure variable as follows: “A binary exposure variable captured whether a complete ban on flavoured tobacco product sales was in effect in the respondent’s district on January 1 of the survey year.”[1] If Friedman had intended to treat the period immediately after July 21 2018 as the “after” period, why had she not selected July 21 of each year as the cut-off date for indicating exposure to the policy effects? It seems apparent that Friedman chose the January 1, 2019 as the cut-off for “after” data because she knew this was the enforcement date and she assumed wrongly that the YRBSS data were collected after January 1, 2019. This is evident in her own response[2] to a critique[3] of her paper as we already noted in our previous response.[4]
Friedman states that “the official/legislated effective date are used to ensure that resulting estimates capture unconfounded responses to the policy change.” Again, if this approach made sense in the specific San Francisco case, why did Friedman use January 1, 2019 in her paper? Perhaps because it simply doesn’t make sense to attribute a policy’s effects before the policy is actually implemented. Similarly, the use of enforcement date rather than...
NOT PEER REVIEWED
On March 17th, 2021, Tobacco Control published a paper online revealing that the 2019 wave of the Youth Risk Behavior Surveillance System (YRBSS) in San Francisco was fielded in the fall of 2018, as opposed to spring of 2019 as is typical for that survey. [1] On March 21st 2022, I received confirmation from San Francisco’s YRBSS site coordinator that the 2019 wave was fielded from November 5th, 2018 to December 14th, 2018. I appreciate Liu and colleagues bringing this to light. However, their claim that this information invalidates the findings from my 2021 JAMA Pediatrics paper [2] —linking San Francisco’s ban on sales of flavored tobacco and nicotine products to increases in youth cigarette smoking—is both methodologically and historically inaccurate: it overlooks both the assumptions required for difference-in-differences research designs and the full timeline of San Francisco’s flavor ban implementation.
In its simplest form, a difference-in-difference (DD) analysis of a particular policy compares outcomes in jurisdictions that did vs. did not adopt the policy, before vs. after that policy officially went into effect (See Figure at https://figshare.com/articles/figure/Figure_1_BasicDDExplanation_pdf/203...). If time-trends in the adopting and non-adopting jurisdictions’ outcomes were parallel in the pre-policy period, the non-adopters’ trends are c...
Show MoreNOT PEER REVIEWED
If PMI attempted to profit from the “EVALI” scaremongering they could only do so because of the blatantly dishonest reporting of that issue by federal authorities, activist academics, tobacco control organisations and the media who quote them with question. It was obvious as early as August 2019 that the lung injuries were caused by black market THC cartridges cut with vitamin E acetate and not nicotine containing e-cigarettes and the CDC eventually came to the same conclusion. Yet activists in positions of authority continue to link the injuries with nicotine vaping, thus providing a fertile ground of misinformation in which such marketing campaigns can flourish.
NOT PEER REVIEWED
A brief review of this ‘Industry Watch’ article alleging heated tobacco product advertising through an earned media approach highlights significant methodological errors that are serious enough to invalidate the article’s conclusions, including its title. The authors allege that Philip Morris International (PMI) used the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak to promote IQOS in September 2019 and the weeks that followed. Using the authors’ own tool (TobaccoWatcher.org), we replicated their search strategy and revealed several fundamental and concerning errors in the authors’ analysis.
They report a rise in news stories mentioning IQOS on and after 25th September 2019, and falsely attribute this rise to an article published on our website on 24th September 2019, which they also falsely describe as a “press release”, despite it never being published through a press release distribution service. Our analysis shows that the authors failed to consider several confounding and unrelated events that caused the rise in news coverage of both IQOS and EVALI during the time period in question and which can be found by replicating the authors’ search strategy in TobaccoWatcher.org.
For example, on 25th September 2019, Philip Morris International (PMI) issued a single press release via Business Wire (1) entitled “Philip Morris International Inc. and Altria Group, Inc. End Merger Discussions” (PMI/Altria Annou...
Show MoreNOT PEER REVIEWED
On behalf of my co-authors, we thank Dr. Mishra for taking the time to comment on our paper examining reporting of trials registered by Juul Labs Inc.
We do not doubt that Juul has submitted the results from all of these studies to the FDA as part of their Premarket Tobacco Product Application (PMTA). Unfortunately, this step does not ensure the full results of these trials will be made available to the public, clinicians, and other key stakeholders.
We also acknowledge and appreciate that since conducting our analysis in August 2020, two additional studies, of the five we examined, have appeared in the peer reviewed literature (published in December 2020 and January 2021)[1,2] and additional results may become available through other methods. We understand that these publications may contain additional outcome reporting compared to the posters we assessed. We welcome any and all additional results disclosures from Juul Labs. That said, we believe the risk of outcome reporting bias remains and is, unfortunately, not addressed in Juul’s comment despite being a major facet of our paper. The one publication that was available as of our analysis date excluded 4 of the 19 prespecified secondary outcomes without comment and another 5 outcomes had clear issues in their reporting compared to the registered outcomes. While we have not conducted a detailed assessment of the two new publications, it is apparent that journal publication does no...
Show MoreNOT PEER REVIEWED
I write on behalf of Juul Labs in response to “E-cigarette manufacturers’ compliance with clinical trial reporting expectations: a case series of registered trials by Juul Labs.” We appreciate the public health community’s interest in understanding the Company’s research. However, we want to clarify the article’s reporting on Juul Labs’ research program and publications to date, as well as our legal obligations to report the results of clinical investigations.
First, our studies were conducted in anticipation of our submission of a Premarket Tobacco Product Application (PMTA) to the U.S. Food and Drug Administration (FDA). As such, raw data, results, and statistical code underlying our analyses of all ten registered Juul Labs studies referenced in the Tobacco Control paper have been submitted to FDA (1)
In addition, we have been working on sharing our data in publications and posters. In fact, by the end of this month, we will have published or presented data on nine of the ten studies, in addition to having provided the clinical results to FDA. Of our ten registered studies, we have published the results of four (2, 3, 4, 5) submitted the results of an additional two studies for peer review, presented the results of two as posters (6,7) and have the results of one accepted for presentation this summer. We appreciate that the authors of the Tobacco Control paper may have drafted their article months ago, but note that there...
Show MoreNOT PEER REVIEWED
A review of this study has been published by the target of its criticism, the tobacco company Philip Morris International (PMI), via the post-publication review server Qeios [1]
The main finding of this study, and the allegation raised in its title, is that PMI cynically used an outbreak of lung injuries in the United States (initially but incorrectly attributed to nicotine vaping) to promote its heated tobacco product, iQOS. Heated tobacco products are one alternative to vaping for those looking for a safer alternative to smoking. On 24th September 2019, PMI published an information notice about its products in response to the lung injury outbreak. The authors assert that PMI was trying to gain commercial publicity from a health crisis: a serious allegation. But the allegation appears to be based on a major error by the authors.
The study used a "fully automated media analysis engine" to count stories that mention iQOS around that time, showing that there were considerably more than usual. On this basis, the authors concluded that PMI's unethical promotional gambit had worked. However, the day after PMI allegedly disreputably sought publicity for iQOS, the company also issued a press release disclosing that merger negotiations with the American tobacco company, Altria, had ceased. PMI and Altria have a joint marketing agreement for iQOS in the United States. The end of merger talks would be big news in the business pre...
Show MoreNOT PEER REVIEWED
We thank Dr. Moira Gilchrist (1) for her careful attention to our work (2). Gilchrist argues our principal findings were erroneous and any change in news coverage of IQOS and the e-cigarette, or vaping, product use associated lung injury (EVALI) outbreak were confounded by other Philip Morris International (PMI) media materials and not those specifically discussing EVALI and IQOS (3) which we attributed our findings to. However, a deeper inspection of this argument suggests our original findings and conclusions remain plausible.
Tobacco Watcher is a dynamic resource with continuous data collection and processing. Thus, the results of analyses on the platform can vary over time. On June 10, 2021 we replicated our analysis. After correcting an error that the PMI’s materials on EVALI and IQOS (3) was initially published on 24 September 2021 (not 25 September) the principal finding is unchanged. News coverage mentioning both “IQOS” and EVALI (i.e., including the terms ‘vaping’ and ‘illness’) reached an all-time high immediately after PMI published materials about EVALI and IQOS on their website. Thirty days prior to PMI posting this material (August 25th through September 23rd) 2.0 news stories per day matched our search compared to 12.8 for the 30 days after their publication (September 24th through October 23rd), with 384 news reports matching our keyword search for the latter period. Our original assertion that there were 14 duplicate articl...
Show MoreNOT PEER REVIEWED
The authors’ response published on 14 July 2021 is far from satisfactory and implausibly asserts that “Our original findings and conclusions remain plausible” [1]
The original study [2] uses a media analysis to make a claim that a statement made by the tobacco company Philip Morris International (PMI) about an outbreak of lung disease in the US [3] was a marketing ploy for its heated tobacco product, iQOS. At the time, the lung injury outbreak was falsely attributed by many to nicotine vaping. Heated tobacco products are an alternative to nicotine vaping for smokers looking for a low-risk alternative to smoking.
I will now list some of the problems with this claim.
1. The research findings do not support the headline claim
The study title contains a strong and unqualified assertion of cynical opportunism on the part of the company. The new formulation that findings "remain plausible" does not justify the confidence in the assertion made in the title. "Plausible" is a reasonable basis for choosing a hypothesis to investigate, but a far from sufficient basis for drawing an aggressive conclusion. The authors do not seem to dispute the technical or factual accuracy of the statement about iQOS and EVALI made by PMI. Their allegation is about malign motives and, as such, it should be a cause for caution and a high standard of evidence.
2. No specific articles were provided to substantiate the...
Show MoreNOT PEER REVIEWED
Replication attempts are one of the self-correcting mechanisms of science, and we thank the Authors for their response to our concerns and their attempt to replicate aspects of their study [1]. Regrettably, they have failed to adequately address the central point raised in our letter of 23rd April 2021, namely that the title and conclusions of their original Article are patently invalid and have no basis in fact or evidence [2]. Instead of strengthening their argument in support of the Article’s findings and conclusions, the Authors’ response considerably weakens them. Strikingly, the Authors reveal several new and serious issues and yet maintain that their “principle finding is unchanged”.
Methodological Problems:
The Authors acknowledge that they were unable to replicate an important aspect of their original analysis, namely that a Philip Morris International (PMI) News Article [3] published on its website (falsely described as a “press release”) was “republished […] in 14 additional news outlets”. In their response, they note that “Our original assertion that there were 14 duplicate articles is not supported by our replication analysis”. This failure to replicate a key finding—in their own proprietary database, which several of them co-developed—is concerning. The Authors provide no explanation for the irregularity. Notably, on 20th April 2021, we were able to source these 14 articles in Tobacco Watcher since they were clearly mar...
Show MorePages