Background The rise of the popular e-cigarette, JUUL, has been partly attributed to various teen-friendly e-liquid flavours offered. However, the possible health risks associated with each e-liquid flavour still remain unclear. This research focuses on the possible associations between JUUL flavours and health symptoms using social media data from Reddit.
Methods Keyword filtering was used to obtain 5,746 JUUL flavour-related posts and 7927 health symptom-related posts from June 2015 to April 2019 from Reddit. Posts from September 2016 to April 2019 were used to conduct temporal analysis for nine health symptom categories and the 8 JUUL flavours. Finally, associations between the JUUL flavours and health symptom categories were examined on the user level using generalised estimating equation models.
Results According to our temporal analysis, Mango and Mint were the most discussed JUUL flavours on Reddit. Respiratory and throat symptoms were the most discussed health issues together with JUUL on Reddit over time. Respiratory symptoms had potential associations with the Mango, Mint and Fruit JUUL flavours. Digestive symptoms had a potential association with the Crème flavour, psychological symptoms had a potential association with the Cucumber flavour, and cardiovascular symptoms had a potential association with the tobacco flavours.
Conclusions Mango and Mint were the two most mentioned JUUL flavours on Reddit. Certain JUUL flavours were more likely to be mentioned together with certain categories of health symptoms by the same Reddit users. Our findings could prompt further medical research into the health symptoms associated with different e-liquid flavours.
- global health
- public opinion
Data availability statement
Data may be obtained from a third party and are not publicly available. The data was obtained from the pushshift.io Reddit comments directory, which contains Reddit posts and comments from December 2005 to April 2019.
Statistics from Altmetric.com
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.
E-cigarette development was promoted in the early 2000s as an effective tool for smoking cessation. However, in recent years, there has been an alarming amount of e-cigarette use, especially among youth in the USA who have never used traditional cigarettes.1 Flavour variety of e-liquid, the main component in e-cigarettes, has been cited as one of the main reasons for the increase in e-cigarette use.2 E-liquid comes in over 7000 flavours which attract users because they each provide unique experiences.3 Although proponents of flavourings say that they help adult smokers quit,4 e-liquid flavours often contain flavour chemicals that have potential risks to users. Many of the chemicals have been labelled as generally safe for ingestion by the Flavor Extracts Manufacturers Association,5 but less is known about their health effects when heated and inhaled by e-cigarette users. Recent studies have focused on discovering which flavours and flavour chemicals are cytotoxic when inhaled through e-cigarette aerosol. A study found that menthol, coffee and strawberry flavours had a significant impact on the overall cytotoxicity of e-cigarette products.6 Another study tested a large variety of e-liquids and discovered that cytotoxic chemicals such as vanillin, ethyl maltol, ethyl vanillin and menthol were the most frequently found flavour chemicals.7 While the cytotoxicity of certain flavour chemicals has been established, little is known about the relationship between e-liquid flavours and health symptoms.
A new generation of e-cigarettes has emerged known as ‘pod mod devices’ which are popular among youth due to their discrete, sleek designs and easily-operational, high-tech features.8 JUUL, a USB drive-shaped pod mod device that debuted in mid-2015, has become dominant in the e-cigarette market, currently controlling a considerable amount of the USA e-cigarette market share and expanding worldwide.9 Although JUULs are currently being marketed as an alternative to smoking, studies have shown that underage users ranging from 13 to 21 years old use JUULs frequently.10 JUUL revolutionised the industry by using cartridges pre-filled with e-liquid containing nicotine salts, flavourings and other chemicals.11 JUUL pods contain nicotine salts instead of free-base nicotine used in most e-cigarettes.12 Nicotine salts allow JUUL to pack their pods with 59 mg/mL of nicotine which is high for e-liquid standards.13 JUUL Pods come in eight different flavours: Mango, Mint, Menthol, Fruit, Crème, Classic Tobacco, Virginia Tobacco and Cucumber.13 We chose to analyse JUUL flavours due to the limited flavour selection and JUUL’s dominance of the e-cigarette market.
In this study, we proposed the use of social media to deduce possible associations between different JUUL flavours and health issues, as well as track JUUL-related flavour and health trends over time. JUULs and the experiences surrounding them are heavily discussed on social media,14 which is why we used social media data for our analysis. Social media is an important source of information that can be used to monitor behaviours within a community and ascertain views of the general population in real time.15 Studies have also shown that web data can more quickly and comprehensively reveal the prevalence of health outcomes compared to surveys.16 We collected data from Reddit which is one of the biggest social media platforms. Reddit operates like a forum which is divided into communities called ‘subreddits’ each covering a specific topic.17 Reddit users can post and comment on others’ posts which promotes discussion and dissemination of information.17 Reddit generally contains less product advertisements and promotions, and more self-reported discussions, making it a good source for reflecting the opinions and experiences of users. Several studies have used Reddit datasets to conduct public health research related to topics like the Ebola outbreak and mental health.18 19
In addition, there have been a growing body of studies using Reddit and other social media such as Twitter and Instagram to analyse e-cigarette marketing, user opinions and flavours of e-liquid.15 20–24 A few studies have used Reddit to analyse JUUL-specific posts in order to understand user perceptions, common user demographics and reasons for use.10 25 26 However, most of these studies have not focused on health issues related to e-cigarettes or JUULs. Li et al is the only study that uses Reddit to qualitatively analyse the potential associations between health symptoms and e-liquid components (such as PG/VG ratios, flavours and nicotine levels).20 In this work, we extend previous studies by conducting quantitative statistical analysis to determine possible associations between JUUL flavours and certain health problems, using post data from Reddit. Considering the potential changes in the prevalence of different JUUL flavours over time, we conducted temporal analysis to visualise longitudinal change in flavour and health trends related to JUUL. This study’s findings could provide insights about the potential health effects of JUUL flavours which could accelerate further medical research.
Data collection from Reddit
The data were obtained from the pushshift.io Reddit comments directory which contains Reddit posts and comments from December 2005 to April 2019.27 Data from 1 January 2013 to 30 April 2019 was downloaded and processed, and e-cigarette-related posts were obtained by filtering posts with the following e-cigarette-related keywords: ‘e-cig’, ‘e-cigs’, ‘ecig’, ‘ecigs’, ‘electroniccigarette’, ‘ecigarette’, ‘ecigarettes’, ‘vape’, ‘vapers’, ‘vaping’, ‘vapes’, ‘e-liquid’, ‘ejuice’, ‘eliquid’, ‘e-juice’, ‘vapercon’, ‘vapeon’, ‘vapefam’, ‘vapenation’ and ‘juul’. We obtained the JUUL related posts by further filtering with specific JUUL-related keywords: ‘juul’, ‘juuling’, ‘juuled’, ‘juuls’ and ‘juuler’. Posts that contained irrelevant mentions (ie, posts mentioning a game designer with the last name ‘Juul’) were removed by filtering out posts that contained targeted keywords. Emojis, URLs and non-English posts were removed during the data cleaning. The final JUUL-related dataset contained 106 183 posts, spanning from 6 October 2013 to 30 April 2019.
Flavours and health categorisation
The JUUL flavours were used as keywords to obtain the subset of flavour-related JUUL data. The flavour subset contained 5746 posts and 3843 Redditors. Similarly, we obtained the health-related dataset using 134 keywords related to health problems and symptoms, which were selected based on user-reported symptoms of e-cigarette use compiled before.28 The keywords were then grouped into nine categories that were similar to the 12 categories created in a previous study.28 The categories are Throat, Respiratory, Psychological, Neurological, Mouth, Digestive, Cardiovascular, Cancer and Other (online supplementary table 1). The Mouth & Throat category28 was separated to better reflect different symptoms. Certain Neurological keywords were split into a new Psychological category, and a new Cancer category was added to gauge Reddit user perceptions of cancer in relation to JUUL. Certain health keywords that linguistically appeared within other health keywords, such as “heart” and “heart attack”, were counted separately. All tenses of each keyword were included to avoid misclassification (online supplementary table 1). The final health subset contained 7927 posts and 5734 Redditors. All posts were tokenised using Tweet Tokenizer in the Natural Language Processing Toolkit29 before keyword filtering to obtain the best accuracy.
Monthly JUUL-related posts from September 2016 to April 2019 were counted to analyse the overall trend of discussions about JUUL. Data before September 2016 was not included because there were less than five posts per month discussing JUUL flavours and health problems. Similar trend plots were created for monthly JUUL post counts for each flavour and health category. Proportion trend plots were created for the flavours and health categories by dividing each monthly count by the total JUUL post count for that month.
To explore potential associations between JUUL flavours and health symptom categories being mentioned by the same users, we first extracted single-flavour users, which we defined as users that only mentioned one flavour among all of their posts, to preserve independence between flavours. We were able to extract a set of 609 single-flavour users. Then, we examined all posts to determine whether the users in this set also mentioned keywords from the health categories. The tobacco flavours, Classic Tobacco and Virginia Tobacco, were combined into a “Tobacco” category for this analysis.
Generalised estimating equation (GEE) models with logit link functions and compound symmetry variance-covariance matrices were used to analyse the potential associations between JUUL flavours and health categories being mentioned by the same users. The GEE model is commonly used to analyse correlated outcomes.30 31 If a user mentioned multiple health categories, each flavour and health category pair was regarded as a unique observation. Thus, observations from the same user are likely to be correlated. Compound symmetry variance-covariance structure assumes observations from the same user have the same correlation. The logit link function converts the values from the model to the scale of a probability,32 which is how we obtained the probabilities of the same user mentioning certain health categories along with specific JUUL flavours. The parameters in the logit link function were calculated using quasi-likelihood functions. Tukey’s method was used to adjust for multiplicity in pairwise comparisons between flavours. The probability of certain health categories being associated with each flavour in the Reddit data was calculated from estimated parameters in the GEE model. The estimated probability denotes the chance that a flavour and health symptom category are mentioned by the same user. Pairwise comparisons between flavours were used to find significant differences in the probabilities of association. A heatmap was created to better visualise the probabilities of association between JUUL flavours and health categories. The GEE models were conducted using the proc genmod procedure in SAS V.9.4 (SAS Institute Inc., Cary, NC). All tests were two-sided with a significance level of 5%.
Temporal analysis of JUUL pod flavours
The number of JUUL-related posts on Reddit were counted for each month from April 2016 to April 2019, illustrating an exponential increase in JUUL-related posts over time (1online supplementary figure 1). There was an increase in the slope of the curve around June 2017, which continued until December of 2018. Since December 2018, the number of JUUL posts has remained steady slightly above 8000 posts per month (online supplementary figure 1).
The most mentioned JUUL flavour on Reddit overall was Mango, followed by Mint. The least mentioned flavours were Virginia Tobacco and Classic Tobacco (online supplementary table 2). This is consistent with the results from our temporal analysis (figure 1). In October 2018, there was a large spike (orange arrow) in the trends of several JUUL pod flavours.
Although the posts for each JUUL flavour were increasing over time, the proportion of posts for each flavour after normalising based on total JUUL posts remained relatively constant with some fluctuations (figure 1B). The red arrow in figure 1B indicates a sudden growth in the proportion of posts mentioning the Mango pod beginning November 2016 and growing to a peak just after March 2017. The purple arrow in figure 1B, indicates a peak in the trends of the Cucumber and Menthol flavours. Overall, the rank of the JUUL flavours discussed on Reddit has remained constant.
Temporal analysis of health categories
Considering the possible association of e-cigarettes with health conditions, we examined health symptoms frequently mentioned together with e-cigarettes on social media. There were nine health categories based on symptoms associated with JUUL on Reddit (online supplementary table 1). Each health category had several keywords that were the main contributors to the overall trend of that health category (table 1). The Respiratory category had the greatest number of posts (n=2535) and its main keywords were ‘lung’ (n=1766), ‘cough’ (n=630) and ‘popcorn lung’ (n=154). Other prominent categories were the Throat category that included ‘throat hit’ (n=1557) and ‘throat’ (n=770), the Psychological category (n=884) that included ‘anxiety’ (n=405), ‘stress’ (n=288) and ‘depression’ (n=197), and the Cardiovascular category (n=561) that included ‘heart’ (n=254) and ‘blood pressure’ (n=66) (table 1).
The number of posts mentioning each health category on Reddit has been increasing over time (figure 2A). The Respiratory and Throat health categories were mentioned the most frequently over time, while the Digestive category was mentioned the least. After adjusting for the overall JUUL-related post count, the proportion of posts related to the Throat category was decreasing over time (figure 2B). The proportion of posts related to the Respiratory category was slightly increasing starting from late 2018. The red arrow in figure 2A,B indicate the same peak in the Respiratory and Cancer health categories in early 2018.
Potential associations between JUUL pod flavours and health categories
The probability of each JUUL flavour being mentioned with different health categories by the same user was calculated in order to examine which flavours might be associated with health symptom categories on Reddit (figure 3). The Throat category had relatively high probabilities of association with all the JUUL flavours. The Menthol flavour had the highest probability of association with the Throat category (0.326), followed by Mint (0.311) and Mango (0.299) flavours. The Fruit flavour had a relatively higher probability of association with the Respiratory category than other flavours. In the Psychological category, the Cucumber flavour had a relatively high probability of association (0.130) compared to other JUUL flavours. In the Digestive category, the Crème flavour had a higher probability of association (0.107) compared to other flavours. In the Cardiovascular category, the combined tobacco flavours had a higher probability of association (0.161) compared to other flavours.
Figure 4 shows the results from pairwise comparisons between each flavour’s probability of association with health categories. Five out of the nine health categories had statistically significant associations with e-liquid flavours. Mango, Mint and Fruit flavours had significantly higher probabilities of association with the Respiratory category than the Menthol flavour (figure 4A). The Crème flavour had a higher probability of association with the Digestive category compared to Mango and Menthol flavours (figure 4B). Compared to the Mango flavour, the Cucumber flavour had a higher probability of association with the Psychological category (figure 4C). The tobacco flavours had a higher probability of association with the Cardiovascular Category than Menthol, Mint and Mango flavours (figure 4D). Reddit users were more likely to post about the tobacco, Fruit and Mango flavours with cancer-related topics (figure 4E). All JUUL flavours had relatively high but similar probabilities of association with the Throat category (figure 4F).
This study provides valuable insights about the potential associations between specific JUUL flavours and categories of health problems. Through temporal analysis, we found that the Mango flavour was consistently the most mentioned JUUL flavour over time, and the Respiratory category was mentioned the most among all examined health categories over time. The association analysis between JUUL flavours and health categories indicated that some JUUL flavours have a greater potential association with certain health problems than others; for example, the JUUL Fruit, Mango and Mint flavours were more likely to be associated with the Respiratory category than the Menthol flavour. This study demonstrates that discussion forums like Reddit could be a good resource for supporting previous research findings and catalysing new research that is based on user experiences and opinions.
JUUL was a good candidate for this analysis because the pods contain the same Propylene Glycol (PG) to Vegetable Glycerin (VG) ratio.13 The nicotine level of JUUL has remained constant at 5%, until the company released new 3% pods in August 2018.11 Since the majority of our data is from before August 2018, we assumed that the nicotine level and PG/VG ratio were consistent across all JUUL flavours. This strengthens our results, as differences in probabilities of association for certain flavours could be attributed more to the different flavour chemicals.
We found that the Respiratory category had greater probabilities of association with the Fruit, Mango and Mint flavours. This is consistent with a study showing higher concentrations of cytotoxic flavour chemicals (vanillin, ethyl maltol, benzyl alcohol) in the Mango and Mint flavours, which could indicate their possible association with Respiratory symptoms.33 The Fruit flavour was found to contain menthol and benzaldehyde PG acetals, which are cytotoxic in aerosol and could also lead to respiratory symptoms.33 On further qualitative analysis of posts in the Respiratory category, we noticed that posters mentioned that they were coughing or had aching lungs because of the JUUL (table 1). However, some posters also described that their lungs felt better when they used the JUUL, in comparison to smoking. Therefore, this suggests that symptom descriptions making up this potential association between Fruit, Mango or Mint users and Respiratory symptoms could be negative, or positive in comparison to smoking.
The combined tobacco flavours had the highest probability of association with the Cardiovascular category. Both Virginia Tobacco and Classic Tobacco have low concentrations of flavour chemicals, with Classic Tobacco containing low levels of benzyl alcohol and Virginia Tobacco containing negligible levels of any flavour chemicals.33 Therefore, cardiovascular symptoms might be due to high nicotine levels in the tobacco-flavoured JUUL pods. In addition, most users of the tobacco flavours switched from smoking or are dual users, because the tobacco flavours have a similar taste to combustible cigarettes.34 These users could have developed these symptoms from traditional cigarette smoking before switching to JUUL.
The Crème flavour had the highest probability of association with the Digestive category. The Crème flavour contains high levels of vanillin, which is cytotoxic in aerosol.33 35 However, there is not much research on how this could affect digestive symptoms. Some users have described trying the Crème flavour and then vomiting, which could be how this association appeared. The Cucumber flavour had the highest probability of association with the Psychological category. Sentiments about psychological symptoms varied. Many posts wrote about anxiety relief after using the JUUL, but a large portion stated that JUUL increased their anxiety. However, research has shown that although nicotine users believe smoking eases their anxiety, this is likely due to their nicotine addiction.36 This may suggest that Cucumber’s possible association relates to its addiction potential. The Cucumber flavouring is made up of mostly menthol, which has been shown to increase nicotine uptake in the blood.37
The Cancer category was created to see user perceptions about cancer on Reddit and what flavours users mention with the hypothetical notion that e-cigarettes could increase the risk of cancer. Cancer develops over time, making it difficult to track on social media, and many of the claims could be inaccurate. Nevertheless, Fruit, Tobacco and Mango had high probabilities of association with the Cancer category.
Based on our temporal analysis of JUUL flavours discussed on Reddit, Mango was consistently the most mentioned flavour over time, followed by Mint. This is consistent with a previous JUUL-related study on Reddit, which counted the total mentions for each flavour.25 A survey study also showed that among teens aged 15–17 years, Mango and Mint were used more often in the past 30 days.38 The tobacco flavours had few mentions in our study, which is consistent with the survey study.38 This illustrates that Reddit is a good indicator of flavour use frequency by youth and young adults. The similarity to the youth survey study could indicate that the population of Reddit users discussing JUUL tend to be younger. This is supported by a study which showed that the age of Redditors participating in JUUL subreddit discussions ranged from 13 to greater than 21 years old.10
The popularity of the Mango and Mint flavours also parallels the flavours of e-juices that are generally the most popular. According to a study about e-cigarette flavours, the most popular flavours are in the Fruit category, followed by the Crème category and Menthol category.39 The eight flavours in the present study seem to be broadly representative of the e-cigarette flavour categories, as Mango and Fruit represent the Fruit category, Crème represents the Crème category, Mint and Menthol represent the Menthol category, Classic Tobacco and Virginia Tobacco represent the Tobacco category, and Cucumber potentially represents the Sweet category. Therefore, our results seem to confirm more general e-cigarette research.
In our temporal analyses of JUUL flavours and health categories, there were no large shifts in the rank of flavours or the perception of health symptoms, but there were certain events that caused fluctuation in the data. The Mango flavour had a sudden growth in mentions (red arrow in figure 1B) which was most likely due to the first release of Mango pods to stores.40 The Cucumber and Menthol flavours also had a peak (purple arrow in figure 1B) after they were released in October 2017.41 Several major peaks corresponded with news events related to JUUL that were heavily discussed. The spike in figure 2 (orange arrow) was due to discussion regarding JUUL suspending its retail sales of Mango, Crème, Cucumber and Fruit to prevent underaged teenagers from obtaining flavour pods.42 The peak in figure 3 (red arrow) came from a tweet that was spread by young adults about several peers being diagnosed with lung cancer. The frenzy originated with a viral tweet that contained screenshots of group chat messages talking about friends being diagnosed with cancer. This tweet was eventually debunked by medical professionals and JUUL Labs, which explains the decrease in mentions shortly after.43 These instances show that Reddit is a good indicator of events over time and more importantly, could be used in a predictive capacity to forecast future rises and falls in certain topics.
There were several limitations in this study. First, solid correlation or causal relationships between the JUUL flavours and health symptoms cannot be established because the symptoms and JUUL flavours are just mentioned by users on social media; therefore, the posters might not necessarily have these symptoms or use these JUUL flavours. However, social media data can provide timely and reasonable estimates of health prevalence and potential associations of e-cigarette flavours with health symptoms. In our Reddit dataset, many users were seeking advice and sharing their experiences, which helped us explore potential associations. We also cannot verify that single-flavour users as we defined them are in fact only using one flavour. Our current analysis only considers the potential association of a single JUUL flavour with health categories mentioned by the same user. We did not distinguish posts expressing JUUL flavours causing or easing certain symptoms from posts expressing irrelativeness of these flavours and symptoms, because we did not conduct syntactical or sentiment analysis for posts. This might cause potential bias in our data analysis but will be considered in future studies. In addition, e-juice flavouring is only one factor that could affect the popularity of a certain e-cigarette product and its association with health categories.44 Therefore, our research could be generalisable to similar pod-mod devices but might not be generalisable to other e-cigarettes with different mechanisms, like box mods or tank devices. Finally, we could not take into account certain demographic information such as age, gender, ethnicity, how long the user has smoked combustible cigarettes, how often the user vapes and so on, because Reddit does not provide this information to the public. In the future, sentiment analysis could be performed to better assess user opinions about JUUL or e-cigarettes with different flavourings. It would also be worthwhile to examine users that switch flavours frequently. On 2 January 2020, the US Food and Drug Administration (FDA) issued a policy to prohibit the sale of unauthorised flavoured cartridge-based e-cigarette products that are appealing to youth, except tobacco and menthol flavours.45 Thus, the prevalence of e-cigarette flavours might change with the enforcement of the FDA policy which will be worth exploring in future studies.
Despite the limitations, this study is the first to quantitatively measure potential associations between JUUL flavours and health symptom categories using JUUL-related Reddit posts. Even with the flavour ban, JUUL is still a popular e-cigarette brand in the USA and is currently expanding its market worldwide with similar flavours. Health symptoms related to JUULing should be further explored, and the results from this study could serve as a springboard for related research about different e-cigarettes, such as disposable e-cigarettes that have similar flavours to JUUL.46 E-cigarettes and their flavourings are associated with certain health issues, as illustrated in this study, and it is necessary to analyse them to provide insights about how to regulate these e-cigarettes and flavours in the future.
What this paper adds
By conducting temporal analysis with normalisation, we show that the discussion about the eight JUUL flavours and nine health symptom categories on Reddit remains relatively constant over time.
Based on user-generated Reddit posts, our results show that different JUUL flavours have different probabilities of association with health symptom categories.
This study illustrates how to conduct quantitative and qualitative analyses of online resources, such as social media, that provide insights into community perspectives and behaviours in real time.
Data availability statement
Data may be obtained from a third party and are not publicly available. The data was obtained from the pushshift.io Reddit comments directory, which contains Reddit posts and comments from December 2005 to April 2019.
Contributors JL, ZX, DL: conceived and designed the study. JL, LC, XL: analysed the data. JL wrote the manuscript. JY, ZX and DL: assisted with interpretation of analyses and edited the manuscript.
Funding Research reported in this publication was supported by the National Cancer Institute of the National Institutes of Health (NIH) and the Food and Drug Administration (FDA) Center for Tobacco Products under Award Number U54CA228110. DL’s time is supported in part by the University of Rochester CTSA award number UL1 TR002001 from the National Center for Advancing Translational Sciences of the National Institutes of Health.
Disclaimer The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH or the FDA.
Competing interests None declared.
Provenance and peer review Not commissioned; externally peer reviewed.