You are here

Article Text

Article menu

Download PDFPDF

Original research
Analysis of potential associations of JUUL flavours with health symptoms based on user-generated data from Reddit
  1. Joyce Luo1,
  2. Long Chen2,
  3. Xinyi Lu3,
  4. Jianbo Yuan2,
  5. Zidian Xie4,
  6. Dongmei Li4
  1. 1 Department of Operations Research & Financial Engineering, Princeton University, Princeton, New Jersey, USA
  2. 2 Department of Computer Science, University of Rochester, Rochester, New York, USA
  3. 3 Goergen Institute for Data Science, University of Rochester, Rochester, New York, USA
  4. 4 Department of Clinical & Translational Research, University of Rochester Medical Center, Rochester, New York, USA
  1. Correspondence to Dr Dongmei Li, Clinical and Translational Science Institute, University of Rochester Medical Center, Rochester, NY 14642-0708, USA; Dongmei_Li{at}urmc.rochester.edu

Abstract

Background The rise of the popular e-cigarette, JUUL, has been partly attributed to various teen-friendly e-liquid flavours offered. However, the possible health risks associated with each e-liquid flavour still remain unclear. This research focuses on the possible associations between JUUL flavours and health symptoms using social media data from Reddit.

Methods Keyword filtering was used to obtain 5,746 JUUL flavour-related posts and 7927 health symptom-related posts from June 2015 to April 2019 from Reddit. Posts from September 2016 to April 2019 were used to conduct temporal analysis for nine health symptom categories and the 8 JUUL flavours. Finally, associations between the JUUL flavours and health symptom categories were examined on the user level using generalised estimating equation models.

Results According to our temporal analysis, Mango and Mint were the most discussed JUUL flavours on Reddit. Respiratory and throat symptoms were the most discussed health issues together with JUUL on Reddit over time. Respiratory symptoms had potential associations with the Mango, Mint and Fruit JUUL flavours. Digestive symptoms had a potential association with the Crème flavour, psychological symptoms had a potential association with the Cucumber flavour, and cardiovascular symptoms had a potential association with the tobacco flavours.

Conclusions Mango and Mint were the two most mentioned JUUL flavours on Reddit. Certain JUUL flavours were more likely to be mentioned together with certain categories of health symptoms by the same Reddit users. Our findings could prompt further medical research into the health symptoms associated with different e-liquid flavours.

  • global health
  • public opinion
  • prevention

Data availability statement

Data may be obtained from a third party and are not publicly available. The data was obtained from the pushshift.io Reddit comments directory, which contains Reddit posts and comments from December 2005 to April 2019.

https://doi.org/10.1136/tobaccocontrol-2019-055439

Statistics from Altmetric.com

    Request Permissions

    If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

    Introduction

    E-cigarette development was promoted in the early 2000s as an effective tool for smoking cessation. However, in recent years, there has been an alarming amount of e-cigarette use, especially among youth in the USA who have never used traditional cigarettes.1 Flavour variety of e-liquid, the main component in e-cigarettes, has been cited as one of the main reasons for the increase in e-cigarette use.2 E-liquid comes in over 7000 flavours which attract users because they each provide unique experiences.3 Although proponents of flavourings say that they help adult smokers quit,4 e-liquid flavours often contain flavour chemicals that have potential risks to users. Many of the chemicals have been labelled as generally safe for ingestion by the Flavor Extracts Manufacturers Association,5 but less is known about their health effects when heated and inhaled by e-cigarette users. Recent studies have focused on discovering which flavours and flavour chemicals are cytotoxic when inhaled through e-cigarette aerosol. A study found that menthol, coffee and strawberry flavours had a significant impact on the overall cytotoxicity of e-cigarette products.6 Another study tested a large variety of e-liquids and discovered that cytotoxic chemicals such as vanillin, ethyl maltol, ethyl vanillin and menthol were the most frequently found flavour chemicals.7 While the cytotoxicity of certain flavour chemicals has been established, little is known about the relationship between e-liquid flavours and health symptoms.

    A new generation of e-cigarettes has emerged known as ‘pod mod devices’ which are popular among youth due to their discrete, sleek designs and easily-operational, high-tech features.8 JUUL, a USB drive-shaped pod mod device that debuted in mid-2015, has become dominant in the e-cigarette market, currently controlling a considerable amount of the USA e-cigarette market share and expanding worldwide.9 Although JUULs are currently being marketed as an alternative to smoking, studies have shown that underage users ranging from 13 to 21 years old use JUULs frequently.10 JUUL revolutionised the industry by using cartridges pre-filled with e-liquid containing nicotine salts, flavourings and other chemicals.11 JUUL pods contain nicotine salts instead of free-base nicotine used in most e-cigarettes.12 Nicotine salts allow JUUL to pack their pods with 59 mg/mL of nicotine which is high for e-liquid standards.13 JUUL Pods come in eight different flavours: Mango, Mint, Menthol, Fruit, Crème, Classic Tobacco, Virginia Tobacco and Cucumber.13 We chose to analyse JUUL flavours due to the limited flavour selection and JUUL’s dominance of the e-cigarette market.

    In this study, we proposed the use of social media to deduce possible associations between different JUUL flavours and health issues, as well as track JUUL-related flavour and health trends over time. JUULs and the experiences surrounding them are heavily discussed on social media,14 which is why we used social media data for our analysis. Social media is an important source of information that can be used to monitor behaviours within a community and ascertain views of the general population in real time.15 Studies have also shown that web data can more quickly and comprehensively reveal the prevalence of health outcomes compared to surveys.16 We collected data from Reddit which is one of the biggest social media platforms. Reddit operates like a forum which is divided into communities called ‘subreddits’ each covering a specific topic.17 Reddit users can post and comment on others’ posts which promotes discussion and dissemination of information.17 Reddit generally contains less product advertisements and promotions, and more self-reported discussions, making it a good source for reflecting the opinions and experiences of users. Several studies have used Reddit datasets to conduct public health research related to topics like the Ebola outbreak and mental health.18 19

    In addition, there have been a growing body of studies using Reddit and other social media such as Twitter and Instagram to analyse e-cigarette marketing, user opinions and flavours of e-liquid.15 20–24 A few studies have used Reddit to analyse JUUL-specific posts in order to understand user perceptions, common user demographics and reasons for use.10 25 26 However, most of these studies have not focused on health issues related to e-cigarettes or JUULs. Li et al is the only study that uses Reddit to qualitatively analyse the potential associations between health symptoms and e-liquid components (such as PG/VG ratios, flavours and nicotine levels).20 In this work, we extend previous studies by conducting quantitative statistical analysis to determine possible associations between JUUL flavours and certain health problems, using post data from Reddit. Considering the potential changes in the prevalence of different JUUL flavours over time, we conducted temporal analysis to visualise longitudinal change in flavour and health trends related to JUUL. This study’s findings could provide insights about the potential health effects of JUUL flavours which could accelerate further medical research.

    Method

    Data collection from Reddit

    The data were obtained from the pushshift.io Reddit comments directory which contains Reddit posts and comments from December 2005 to April 2019.27 Data from 1 January 2013 to 30 April 2019 was downloaded and processed, and e-cigarette-related posts were obtained by filtering posts with the following e-cigarette-related keywords: ‘e-cig’, ‘e-cigs’, ‘ecig’, ‘ecigs’, ‘electroniccigarette’, ‘ecigarette’, ‘ecigarettes’, ‘vape’, ‘vapers’, ‘vaping’, ‘vapes’, ‘e-liquid’, ‘ejuice’, ‘eliquid’, ‘e-juice’, ‘vapercon’, ‘vapeon’, ‘vapefam’, ‘vapenation’ and ‘juul’. We obtained the JUUL related posts by further filtering with specific JUUL-related keywords: ‘juul’, ‘juuling’, ‘juuled’, ‘juuls’ and ‘juuler’. Posts that contained irrelevant mentions (ie, posts mentioning a game designer with the last name ‘Juul’) were removed by filtering out posts that contained targeted keywords. Emojis, URLs and non-English posts were removed during the data cleaning. The final JUUL-related dataset contained 106 183 posts, spanning from 6 October 2013 to 30 April 2019.

    Flavours and health categorisation

    The JUUL flavours were used as keywords to obtain the subset of flavour-related JUUL data. The flavour subset contained 5746 posts and 3843 Redditors. Similarly, we obtained the health-related dataset using 134 keywords related to health problems and symptoms, which were selected based on user-reported symptoms of e-cigarette use compiled before.28 The keywords were then grouped into nine categories that were similar to the 12 categories created in a previous study.28 The categories are Throat, Respiratory, Psychological, Neurological, Mouth, Digestive, Cardiovascular, Cancer and Other (online supplementary table 1). The Mouth & Throat category28 was separated to better reflect different symptoms. Certain Neurological keywords were split into a new Psychological category, and a new Cancer category was added to gauge Reddit user perceptions of cancer in relation to JUUL. Certain health keywords that linguistically appeared within other health keywords, such as “heart” and “heart attack”, were counted separately. All tenses of each keyword were included to avoid misclassification (online supplementary table 1). The final health subset contained 7927 posts and 5734 Redditors. All posts were tokenised using Tweet Tokenizer in the Natural Language Processing Toolkit29 before keyword filtering to obtain the best accuracy.

    Temporal analysis

    Monthly JUUL-related posts from September 2016 to April 2019 were counted to analyse the overall trend of discussions about JUUL. Data before September 2016 was not included because there were less than five posts per month discussing JUUL flavours and health problems. Similar trend plots were created for monthly JUUL post counts for each flavour and health category. Proportion trend plots were created for the flavours and health categories by dividing each monthly count by the total JUUL post count for that month.

    Statistical analysis

    To explore potential associations between JUUL flavours and health symptom categories being mentioned by the same users, we first extracted single-flavour users, which we defined as users that only mentioned one flavour among all of their posts, to preserve independence between flavours. We were able to extract a set of 609 single-flavour users. Then, we examined all posts to determine whether the users in this set also mentioned keywords from the health categories. The tobacco flavours, Classic Tobacco and Virginia Tobacco, were combined into a “Tobacco” category for this analysis.

    Generalised estimating equation (GEE) models with logit link functions and compound symmetry variance-covariance matrices were used to analyse the potential associations between JUUL flavours and health categories being mentioned by the same users. The GEE model is commonly used to analyse correlated outcomes.30 31 If a user mentioned multiple health categories, each flavour and health category pair was regarded as a unique observation. Thus, observations from the same user are likely to be correlated. Compound symmetry variance-covariance structure assumes observations from the same user have the same correlation. The logit link function converts the values from the model to the scale of a probability,32 which is how we obtained the probabilities of the same user mentioning certain health categories along with specific JUUL flavours. The parameters in the logit link function were calculated using quasi-likelihood functions. Tukey’s method was used to adjust for multiplicity in pairwise comparisons between flavours. The probability of certain health categories being associated with each flavour in the Reddit data was calculated from estimated parameters in the GEE model. The estimated probability denotes the chance that a flavour and health symptom category are mentioned by the same user. Pairwise comparisons between flavours were used to find significant differences in the probabilities of association. A heatmap was created to better visualise the probabilities of association between JUUL flavours and health categories. The GEE models were conducted using the proc genmod procedure in SAS V.9.4 (SAS Institute Inc., Cary, NC). All tests were two-sided with a significance level of 5%.

    Results

    Temporal analysis of JUUL pod flavours

    The number of JUUL-related posts on Reddit were counted for each month from April 2016 to April 2019, illustrating an exponential increase in JUUL-related posts over time (1online supplementary figure 1). There was an increase in the slope of the curve around June 2017, which continued until December of 2018. Since December 2018, the number of JUUL posts has remained steady slightly above 8000 posts per month (online supplementary figure 1).

    Supplemental material

    Number of monthly JUUL-related posts on Reddit over time.

    The most mentioned JUUL flavour on Reddit overall was Mango, followed by Mint. The least mentioned flavours were Virginia Tobacco and Classic Tobacco (online supplementary table 2). This is consistent with the results from our temporal analysis (figure 1). In October 2018, there was a large spike (orange arrow) in the trends of several JUUL pod flavours.

    Figure 1
    Figure 1

    (A) Number of monthly posts for each JUUL flavour pod on Reddit over time. (B) Proportion of monthly posts for each JUUL flavour pod on Reddit over time.

    Although the posts for each JUUL flavour were increasing over time, the proportion of posts for each flavour after normalising based on total JUUL posts remained relatively constant with some fluctuations (figure 1B). The red arrow in figure 1B indicates a sudden growth in the proportion of posts mentioning the Mango pod beginning November 2016 and growing to a peak just after March 2017. The purple arrow in figure 1B, indicates a peak in the trends of the Cucumber and Menthol flavours. Overall, the rank of the JUUL flavours discussed on Reddit has remained constant.

    Temporal analysis of health categories

    Considering the possible association of e-cigarettes with health conditions, we examined health symptoms frequently mentioned together with e-cigarettes on social media. There were nine health categories based on symptoms associated with JUUL on Reddit (online supplementary table 1). Each health category had several keywords that were the main contributors to the overall trend of that health category (table 1). The Respiratory category had the greatest number of posts (n=2535) and its main keywords were ‘lung’ (n=1766), ‘cough’ (n=630) and ‘popcorn lung’ (n=154). Other prominent categories were the Throat category that included ‘throat hit’ (n=1557) and ‘throat’ (n=770), the Psychological category (n=884) that included ‘anxiety’ (n=405), ‘stress’ (n=288) and ‘depression’ (n=197), and the Cardiovascular category (n=561) that included ‘heart’ (n=254) and ‘blood pressure’ (n=66) (table 1).

    View this table:
    Table 1

    Top keywords in each health category

    The number of posts mentioning each health category on Reddit has been increasing over time (figure 2A). The Respiratory and Throat health categories were mentioned the most frequently over time, while the Digestive category was mentioned the least. After adjusting for the overall JUUL-related post count, the proportion of posts related to the Throat category was decreasing over time (figure 2B). The proportion of posts related to the Respiratory category was slightly increasing starting from late 2018. The red arrow in figure 2A,B indicate the same peak in the Respiratory and Cancer health categories in early 2018.

    Figure 2
    Figure 2

    (A) Number of monthly posts for each health category on Reddit over time. (B) proportion of monthly posts for each health category on Reddit over time.

    Potential associations between JUUL pod flavours and health categories

    The probability of each JUUL flavour being mentioned with different health categories by the same user was calculated in order to examine which flavours might be associated with health symptom categories on Reddit (figure 3). The Throat category had relatively high probabilities of association with all the JUUL flavours. The Menthol flavour had the highest probability of association with the Throat category (0.326), followed by Mint (0.311) and Mango (0.299) flavours. The Fruit flavour had a relatively higher probability of association with the Respiratory category than other flavours. In the Psychological category, the Cucumber flavour had a relatively high probability of association (0.130) compared to other JUUL flavours. In the Digestive category, the Crème flavour had a higher probability of association (0.107) compared to other flavours. In the Cardiovascular category, the combined tobacco flavours had a higher probability of association (0.161) compared to other flavours.

    Figure 3
    Figure 3

    Heatmap illustrating the probabilities of association between JUUL pod flavours and health categories.

    Figure 4 shows the results from pairwise comparisons between each flavour’s probability of association with health categories. Five out of the nine health categories had statistically significant associations with e-liquid flavours. Mango, Mint and Fruit flavours had significantly higher probabilities of association with the Respiratory category than the Menthol flavour (figure 4A). The Crème flavour had a higher probability of association with the Digestive category compared to Mango and Menthol flavours (figure 4B). Compared to the Mango flavour, the Cucumber flavour had a higher probability of association with the Psychological category (figure 4C). The tobacco flavours had a higher probability of association with the Cardiovascular Category than Menthol, Mint and Mango flavours (figure 4D). Reddit users were more likely to post about the tobacco, Fruit and Mango flavours with cancer-related topics (figure 4E). All JUUL flavours had relatively high but similar probabilities of association with the Throat category (figure 4F).

    Figure 4
    Figure 4

    Bar charts illustrating the probabilities of association between the JUUL pod flavours and health categories, including the (A) respiratory, (B) digestive, (C) psychological, (D) cardiovascular, (E) cancer and (F) throat health categories. There were no significant pairwise comparisons between flavours for the throat, mouth, neurological and other health categories. throat was included because the probabilities of association were relatively high for all the flavours. Error bars represent 95% CI. *P<0.05; **P<0.01.

    Discussion

    This study provides valuable insights about the potential associations between specific JUUL flavours and categories of health problems. Through temporal analysis, we found that the Mango flavour was consistently the most mentioned JUUL flavour over time, and the Respiratory category was mentioned the most among all examined health categories over time. The association analysis between JUUL flavours and health categories indicated that some JUUL flavours have a greater potential association with certain health problems than others; for example, the JUUL Fruit, Mango and Mint flavours were more likely to be associated with the Respiratory category than the Menthol flavour. This study demonstrates that discussion forums like Reddit could be a good resource for supporting previous research findings and catalysing new research that is based on user experiences and opinions.

    JUUL was a good candidate for this analysis because the pods contain the same Propylene Glycol (PG) to Vegetable Glycerin (VG) ratio.13 The nicotine level of JUUL has remained constant at 5%, until the company released new 3% pods in August 2018.11 Since the majority of our data is from before August 2018, we assumed that the nicotine level and PG/VG ratio were consistent across all JUUL flavours. This strengthens our results, as differences in probabilities of association for certain flavours could be attributed more to the different flavour chemicals.

    We found that the Respiratory category had greater probabilities of association with the Fruit, Mango and Mint flavours. This is consistent with a study showing higher concentrations of cytotoxic flavour chemicals (vanillin, ethyl maltol, benzyl alcohol) in the Mango and Mint flavours, which could indicate their possible association with Respiratory symptoms.33 The Fruit flavour was found to contain menthol and benzaldehyde PG acetals, which are cytotoxic in aerosol and could also lead to respiratory symptoms.33 On further qualitative analysis of posts in the Respiratory category, we noticed that posters mentioned that they were coughing or had aching lungs because of the JUUL (table 1). However, some posters also described that their lungs felt better when they used the JUUL, in comparison to smoking. Therefore, this suggests that symptom descriptions making up this potential association between Fruit, Mango or Mint users and Respiratory symptoms could be negative, or positive in comparison to smoking.

    The combined tobacco flavours had the highest probability of association with the Cardiovascular category. Both Virginia Tobacco and Classic Tobacco have low concentrations of flavour chemicals, with Classic Tobacco containing low levels of benzyl alcohol and Virginia Tobacco containing negligible levels of any flavour chemicals.33 Therefore, cardiovascular symptoms might be due to high nicotine levels in the tobacco-flavoured JUUL pods. In addition, most users of the tobacco flavours switched from smoking or are dual users, because the tobacco flavours have a similar taste to combustible cigarettes.34 These users could have developed these symptoms from traditional cigarette smoking before switching to JUUL.

    The Crème flavour had the highest probability of association with the Digestive category. The Crème flavour contains high levels of vanillin, which is cytotoxic in aerosol.33 35 However, there is not much research on how this could affect digestive symptoms. Some users have described trying the Crème flavour and then vomiting, which could be how this association appeared. The Cucumber flavour had the highest probability of association with the Psychological category. Sentiments about psychological symptoms varied. Many posts wrote about anxiety relief after using the JUUL, but a large portion stated that JUUL increased their anxiety. However, research has shown that although nicotine users believe smoking eases their anxiety, this is likely due to their nicotine addiction.36 This may suggest that Cucumber’s possible association relates to its addiction potential. The Cucumber flavouring is made up of mostly menthol, which has been shown to increase nicotine uptake in the blood.37

    The Cancer category was created to see user perceptions about cancer on Reddit and what flavours users mention with the hypothetical notion that e-cigarettes could increase the risk of cancer. Cancer develops over time, making it difficult to track on social media, and many of the claims could be inaccurate. Nevertheless, Fruit, Tobacco and Mango had high probabilities of association with the Cancer category.

    Based on our temporal analysis of JUUL flavours discussed on Reddit, Mango was consistently the most mentioned flavour over time, followed by Mint. This is consistent with a previous JUUL-related study on Reddit, which counted the total mentions for each flavour.25 A survey study also showed that among teens aged 15–17 years, Mango and Mint were used more often in the past 30 days.38 The tobacco flavours had few mentions in our study, which is consistent with the survey study.38 This illustrates that Reddit is a good indicator of flavour use frequency by youth and young adults. The similarity to the youth survey study could indicate that the population of Reddit users discussing JUUL tend to be younger. This is supported by a study which showed that the age of Redditors participating in JUUL subreddit discussions ranged from 13 to greater than 21 years old.10

    The popularity of the Mango and Mint flavours also parallels the flavours of e-juices that are generally the most popular. According to a study about e-cigarette flavours, the most popular flavours are in the Fruit category, followed by the Crème category and Menthol category.39 The eight flavours in the present study seem to be broadly representative of the e-cigarette flavour categories, as Mango and Fruit represent the Fruit category, Crème represents the Crème category, Mint and Menthol represent the Menthol category, Classic Tobacco and Virginia Tobacco represent the Tobacco category, and Cucumber potentially represents the Sweet category. Therefore, our results seem to confirm more general e-cigarette research.

    In our temporal analyses of JUUL flavours and health categories, there were no large shifts in the rank of flavours or the perception of health symptoms, but there were certain events that caused fluctuation in the data. The Mango flavour had a sudden growth in mentions (red arrow in figure 1B) which was most likely due to the first release of Mango pods to stores.40 The Cucumber and Menthol flavours also had a peak (purple arrow in figure 1B) after they were released in October 2017.41 Several major peaks corresponded with news events related to JUUL that were heavily discussed. The spike in figure 2 (orange arrow) was due to discussion regarding JUUL suspending its retail sales of Mango, Crème, Cucumber and Fruit to prevent underaged teenagers from obtaining flavour pods.42 The peak in figure 3 (red arrow) came from a tweet that was spread by young adults about several peers being diagnosed with lung cancer. The frenzy originated with a viral tweet that contained screenshots of group chat messages talking about friends being diagnosed with cancer. This tweet was eventually debunked by medical professionals and JUUL Labs, which explains the decrease in mentions shortly after.43 These instances show that Reddit is a good indicator of events over time and more importantly, could be used in a predictive capacity to forecast future rises and falls in certain topics.

    There were several limitations in this study. First, solid correlation or causal relationships between the JUUL flavours and health symptoms cannot be established because the symptoms and JUUL flavours are just mentioned by users on social media; therefore, the posters might not necessarily have these symptoms or use these JUUL flavours. However, social media data can provide timely and reasonable estimates of health prevalence and potential associations of e-cigarette flavours with health symptoms. In our Reddit dataset, many users were seeking advice and sharing their experiences, which helped us explore potential associations. We also cannot verify that single-flavour users as we defined them are in fact only using one flavour. Our current analysis only considers the potential association of a single JUUL flavour with health categories mentioned by the same user. We did not distinguish posts expressing JUUL flavours causing or easing certain symptoms from posts expressing irrelativeness of these flavours and symptoms, because we did not conduct syntactical or sentiment analysis for posts. This might cause potential bias in our data analysis but will be considered in future studies. In addition, e-juice flavouring is only one factor that could affect the popularity of a certain e-cigarette product and its association with health categories.44 Therefore, our research could be generalisable to similar pod-mod devices but might not be generalisable to other e-cigarettes with different mechanisms, like box mods or tank devices. Finally, we could not take into account certain demographic information such as age, gender, ethnicity, how long the user has smoked combustible cigarettes, how often the user vapes and so on, because Reddit does not provide this information to the public. In the future, sentiment analysis could be performed to better assess user opinions about JUUL or e-cigarettes with different flavourings. It would also be worthwhile to examine users that switch flavours frequently. On 2 January 2020, the US Food and Drug Administration (FDA) issued a policy to prohibit the sale of unauthorised flavoured cartridge-based e-cigarette products that are appealing to youth, except tobacco and menthol flavours.45 Thus, the prevalence of e-cigarette flavours might change with the enforcement of the FDA policy which will be worth exploring in future studies.

    Despite the limitations, this study is the first to quantitatively measure potential associations between JUUL flavours and health symptom categories using JUUL-related Reddit posts. Even with the flavour ban, JUUL is still a popular e-cigarette brand in the USA and is currently expanding its market worldwide with similar flavours. Health symptoms related to JUULing should be further explored, and the results from this study could serve as a springboard for related research about different e-cigarettes, such as disposable e-cigarettes that have similar flavours to JUUL.46 E-cigarettes and their flavourings are associated with certain health issues, as illustrated in this study, and it is necessary to analyse them to provide insights about how to regulate these e-cigarettes and flavours in the future.

    What this paper adds

    • By conducting temporal analysis with normalisation, we show that the discussion about the eight JUUL flavours and nine health symptom categories on Reddit remains relatively constant over time.

    • Based on user-generated Reddit posts, our results show that different JUUL flavours have different probabilities of association with health symptom categories.

    • This study illustrates how to conduct quantitative and qualitative analyses of online resources, such as social media, that provide insights into community perspectives and behaviours in real time.

    Data availability statement

    Data may be obtained from a third party and are not publicly available. The data was obtained from the pushshift.io Reddit comments directory, which contains Reddit posts and comments from December 2005 to April 2019.

    Ethics statements

    References

    1. ↵
      2. Barrington-Trimis JL ,
      3. Kong G ,
      4. Leventhal AM , et al
      . E-Cigarette use and subsequent smoking frequency among adolescents. Pediatrics 2018;142:e20180486. doi:10.1542/peds.2018-0486 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30397165
      OpenUrlAbstract/FREE Full Text
    2. ↵
      1. FDA
      . 2018 NYTS data: a startling rise in youth e-cigarette use, 2019. Available: https://www.fda.gov/tobacco-products/youth-and-tobacco/2018-nyts-data-startling-rise-youth-e-cigarette-use#references [Accessed 7 Aug 2019].
    3. ↵
      2. Sandusky L
      . The most popular juice flavors on the market, 2019. Available: https://www.omvapors.com/blog/the-most-popular-juice-flavors-on-the-market/ [Accessed 13 Apr 2020].
    4. ↵
      2. Nedelman M
      . Studies illustrate how marketing, flavors may impact vaping habits among young people, 2019. Available: https://www.cnn.com/2019/10/28/health/vaping-marketing-flavors-study/index.html [Accessed 13 Apr 2020].
    5. ↵
      1. FEMA
      . The safety assessment and regulatory authority to use flavors: focus on e-cigarettes. Available: https://www.femaflavor.org/member-update/safety-assessment-and-regulatory-authority-use-flavors-focus-e-cigarettes [Accessed 14 Aug 2019].
    6. ↵
      2. Leigh NJ ,
      3. Lawton RI ,
      4. Hershberger PA , et al
      . Flavourings significantly affect inhalation toxicity of aerosol generated from electronic nicotine delivery systems (ends). Tob Control 2016;25:ii81–7.doi:10.1136/tobaccocontrol-2016-053205 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27633767
      OpenUrlAbstract/FREE Full Text
    7. ↵
      2. Tierney PA ,
      3. Karpinski CD ,
      4. Brown JE , et al
      . Flavour chemicals in electronic cigarette fluids. Tob Control 2016;25:e10–15.doi:10.1136/tobaccocontrol-2014-052175 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25877377
      OpenUrlAbstract/FREE Full Text
    8. ↵
      2. Fadus MC ,
      3. Smith TT ,
      4. Squeglia LM
      . The rise of e-cigarettes, pod mod devices, and JUUL among youth: factors influencing use, health implications, and downstream effects. Drug Alcohol Depend 2019;201:85–93.doi:10.1016/j.drugalcdep.2019.04.011 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31200279
      OpenUrlPubMed
    9. ↵
      2. Kosman J
      . JUUL E-cigarette Maker Sees its Valuation Top $38B. New York Post, 2019. Available: https://nypost.com/2019/05/17/juul-e-cigarette-maker-sees-its-valuation-top-38b/ [Accessed 8 Aug 2019].
    10. ↵
      2. Zhan Y ,
      3. Zhang Z ,
      4. Okamoto JM , et al
      . Underage JUUL use patterns: content analysis of Reddit messages. J Med Internet Res 2019;21:e13038. doi:10.2196/13038 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31502542
      OpenUrlPubMed
    11. ↵
      2. Capritto A
      . Juul vaporizer: what is it, why are teens addicted and is it safe? CNET, 2019. Available: https://www.cnet.com/news/juul-what-is-it-how-does-it-work-and-is-it-safe/ [Accessed 8 Aug 2019].
    12. ↵
      2. Becker R
      . Why is JUUL Worth $16 Billion? It's More Like a Cigarette Than You Think. The Verge, 2018. Available: https://www.theverge.com/2018/7/3/17529442/juul-vapes-nicotine-electronic-cigarettes-addiction-funding [Accessed 8 Aug 2019].
    13. ↵
      1. JUUL
      . JUULPod basics. Available: https://www.juul.com/learn/pods [Accessed 8 Aug 2019].
    14. ↵
      2. Jackler RK ,
      3. Ramamurthi D ,
      4. Louis-Ferdinand NG
      . Rapid growth of JUUL Hashtags after the company Ceased social media promotion. Stanford research into the impact of tobacco advertising, 2019.
    15. ↵
      2. Allem J-P ,
      3. Dharmapuri L ,
      4. Unger JB , et al
      . Characterizing JUUL-related posts on Twitter. Drug Alcohol Depend 2018;190:1–5.doi:10.1016/j.drugalcdep.2018.05.018 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29958115
      OpenUrlPubMed
    16. ↵
      2. Ayers JW ,
      3. Althouse BM ,
      4. Dredze M
      . Could behavioral medicine lead the web data revolution? JAMA 2014;311:1399–400.doi:10.1001/jama.2014.1505 pmid:http://www.ncbi.nlm.nih.gov/pubmed/24577162
      OpenUrlCrossRefPubMedWeb of Science
    17. ↵
      2. Widman J ,
      3. Nicol W
      . What is Reddit? A beginner’s guide to the front page of the internet, 2019. Available: https://www.digitaltrends.com/web/what-is-reddit/ [Accessed 12 Aug 2019].
    18. ↵
      2. Pavalanathan U ,
      3. De Choudhury M
      . Identity management and mental health discourse in social media. Proc Int World Wide Web Conf 2015;2015:315–21.doi:10.1145/2740908.2743049 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27376158
      OpenUrlPubMed
    19. ↵
      2. Arthur M
      . Reddit: tracking the 2014 Ebola outbreak across the world. Nurs Stand 2014;29:32. doi:10.7748/ns.29.15.32.s37
    20. ↵
      2. Li Q ,
      3. Zhan Y ,
      4. Wang L , et al
      . Analysis of symptoms and their potential associations with e-liquids' components: a social media study. BMC Public Health 2016;16:674. doi:10.1186/s12889-016-3326-0 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27475060
      OpenUrlPubMed
    21. ↵
      2. Lazard AJ ,
      3. Saffer AJ ,
      4. Wilcox GB , et al
      . E-Cigarette social media messages: a text mining analysis of marketing and consumer conversations on Twitter. JMIR Public Health Surveill 2016;2:e171. doi:10.2196/publichealth.6551 pmid:http://www.ncbi.nlm.nih.gov/pubmed/27956376
      OpenUrlPubMed
    22. ↵
      2. Huang J ,
      3. Duan Z ,
      4. Kwok J , et al
      . Vaping versus JUULing: how the extraordinary growth and marketing of JUUL transformed the US retail e-cigarette market. Tob Control 2019;28:146–51.doi:10.1136/tobaccocontrol-2018-054382 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29853561
      OpenUrlAbstract/FREE Full Text
    23. ↵
      2. Sidani JE ,
      3. Colditz JB ,
      4. Barrett EL , et al
      . I wake up and hit the JUUL: analyzing Twitter for JUUL nicotine effects and dependence. Drug Alcohol Depend 2019;204:107500. doi:10.1016/j.drugalcdep.2019.06.005 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31499242
      OpenUrlPubMed
    24. ↵
      2. Ayers JW ,
      3. Leas EC ,
      4. Allem J-P , et al
      . Why do people use electronic nicotine delivery systems (electronic cigarettes)? A content analysis of Twitter, 2012-2015. PLoS One 2017;12:e0170702. doi:10.1371/journal.pone.0170702 pmid:http://www.ncbi.nlm.nih.gov/pubmed/28248987
      OpenUrlPubMed
    25. ↵
      2. Brett EI ,
      3. Stevens EM ,
      4. Wagener TL , et al
      . A content analysis of JUUL discussions on social media: using Reddit to understand patterns and perceptions of JUUL use. Drug Alcohol Depend 2019;194:358–62.doi:10.1016/j.drugalcdep.2018.10.014 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30472576
      OpenUrlCrossRefPubMed
    26. ↵
      2. Liu H ,
      3. Li Q ,
      4. Yao R , et al
      . Analyzing topics of JUUL discussions on social media using a Semantics-assisted NMF model. Paper presented at: 2019 IEEE International Conference on Intelligence and Security Informatics (ISI); 1-3 July, 2019.
    27. ↵
      2. Baumgartner JM
      . pushshift.io, 2018. Available: http://files.pushshift.io/reddit/comments
    28. ↵
      2. Hua M ,
      3. Alfi M ,
      4. Talbot P
      . Health-related effects reported by electronic cigarette users in online forums. J Med Internet Res 2013;15:e59. doi:10.2196/jmir.2324 pmid:http://www.ncbi.nlm.nih.gov/pubmed/23567935
      OpenUrlCrossRefPubMed
    29. ↵
      2. Bird S ,
      3. Klein E ,
      4. Loper E
      . Natural language processing with python. Beijing, Cambridge Mass: O'Reilly, 2009.
    30. ↵
      2. Yin T ,
      3. Bapst B ,
      4. von Borstel UU , et al
      . Genetic analyses of binary longitudinal health data in small low input dairy cattle herds using generalized linear mixed models. Livest Sci 2014;162:31–41.doi:10.1016/j.livsci.2014.01.021
      OpenUrl
    31. ↵
      2. Liang K-YEE ,
      3. Zeger SL
      . Longitudinal data analysis using generalized linear models. Biometrika 1986;73:13–22.doi:10.1093/biomet/73.1.13
      OpenUrlCrossRefWeb of Science
    32. ↵
      2. MacKenzie DI ,
      3. Nichols JD ,
      4. Royle JA , et al
      . Chapter 3 - Fundamental principals of statistical inference. In: MacKenzie DI , Nichols JD , Royle JA , et al. , eds. Occupancy estimation and modeling. 2nd edn. Boston: Academic Press, 2018: 71–111.
    33. ↵
      2. Omaiye EE ,
      3. McWhirter KJ ,
      4. Luo W , et al
      . High-Nicotine electronic cigarette products: toxicity of JUUL fluids and aerosols correlates strongly with nicotine and some flavor chemical concentrations. Chem Res Toxicol 2019;32:1058–69.doi:10.1021/acs.chemrestox.8b00381 pmid:http://www.ncbi.nlm.nih.gov/pubmed/30896936
      OpenUrlCrossRefPubMed
    34. ↵
      1. TruthInitiative
      . JUUL fails to remove all of youth’s favorite flavors from stores, 2018. Available: https://truthinitiative.org/research-resources/emerging-tobacco-products/juul-fails-remove-all-youths-favorite-flavors-stores [Accessed 15 Aug 2019].
    35. ↵
      2. Behar RZ ,
      3. Luo W ,
      4. McWhirter KJ , et al
      . Analytical and toxicological evaluation of flavor chemicals in electronic cigarette refill fluids. Sci Rep 2018;8:8288. doi:10.1038/s41598-018-25575-6 pmid:http://www.ncbi.nlm.nih.gov/pubmed/29844439
      OpenUrlCrossRefPubMed
    36. ↵
      1. Smokefree.gov
      . Anxiety & Smoking. Available: https://smokefree.gov/challenges-when-quitting/cravings-triggers/anxiety-smoking [Accessed 15 Aug 2019].
    37. ↵
      2. Ha MA ,
      3. Smith GJ ,
      4. Cichocki JA , et al
      . Menthol attenuates respiratory irritation and elevates blood cotinine in cigarette smoke exposed mice. PLoS One 2015;10:e0117128. doi:10.1371/journal.pone.0117128 pmid:http://www.ncbi.nlm.nih.gov/pubmed/25679525
      OpenUrlCrossRefPubMed
    38. ↵
      2. McKeganey N ,
      3. Russell C
      . Prevalence of awareness and use of JUUL e-cigarettes in a national probability sample of adolescents in the United States. Am J Health Behav 2019;43:591–605.doi:10.5993/AJHB.43.3.13 pmid:http://www.ncbi.nlm.nih.gov/pubmed/31046889
      OpenUrlPubMed
    39. ↵
      2. Wang L ,
      3. Zhan Y ,
      4. Li Q , et al
      . An examination of electronic cigarette content on social media: analysis of e-cigarette flavor content on Reddit. Int J Environ Res Public Health 2015;12:14916–35.doi:10.3390/ijerph121114916 pmid:http://www.ncbi.nlm.nih.gov/pubmed/26610541
      OpenUrlPubMed
    40. ↵
      2. Srsen I
      . The latest JUUL pod flavor, mango, is back again, 2017. Available: https://ecigarettereviewed.com/juul-mango-pod/ [Accessed 13 Aug 2019].
    41. ↵
      1. JUULLabs
      . Update on limited edition cool cucumber, 2017. Available: https://www.reddit.com/r/juul/comments/770ymq/update_on_limited_edition_cool_cucumber/ [Accessed 13 Aug 2019].
    42. ↵
      2. Kaplan S ,
      3. Hoffman J
      . Juul suspends Selling most e-cigarette flavors in stores. The New York Times, 2018.
    43. ↵
      2. Lacapria K
      . Has ‘JUULing’ Caused Cancer in High School and College Students? 2018. Available: https://www.snopes.com/fact-check/does-juuling-cancer-young-people/ [Accessed 13 Aug 2019].
    44. ↵
      2. Allem J-P ,
      3. Majmundar A ,
      4. Dharmapuri L , et al
      . Insights on electronic cigarette products from reviews on the Reddit forum. Tob Prev Cessat 2019;5:29.doi:10.18332/tpc/111533 pmid:http://www.ncbi.nlm.nih.gov/pubmed/32411892
      OpenUrlPubMed
    45. ↵
      2. Turner T ,
      3. Ban J
      . Available: https://www.drugwatch.com/e-cigarettes/juul-ban/ [Accessed 24 Apr 2020].
    46. ↵
      2. Kaplan S
      . Teens find a big loophole in the new flavored Vaping ban, 2020. Available: https://www.nytimes.com/2020/01/31/health/vaping-flavors-disposable.html [Accessed 24 Apr 2020].

    Footnotes

    • Contributors JL, ZX, DL: conceived and designed the study. JL, LC, XL: analysed the data. JL wrote the manuscript. JY, ZX and DL: assisted with interpretation of analyses and edited the manuscript.

    • Funding Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health (NIH) and the Food and Drug Administration (FDA) Center for Tobacco Products under Award Number U54CA228110. DL’s time is supported in part by the University of Rochester CTSA award number UL1 TR002001 from the National Center for Advancing Translational Sciences of the National Institutes of Health.

    • Disclaimer The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the FDA.

    • Competing interests None declared.

    • Provenance and peer review Not commissioned; externally peer reviewed.