Article Text
PDF
Abstract
Objective To critically assess the methodological characteristics and quality of interventional clinical trials investigating the effects of heated tobacco products (HTPs).
Data sources Web of Science (Core collection and MEDLINE), Scopus, MedRxiv, ClinicalTrials.gov and ICTRP trial databases and transnational HTP manufacturer online publication libraries were searched for clinical trials on HTPs published between January 2010 and April 2022.
Study selection Interventional clinical trials of any design, in which at least one group of adult participants used a currently marketed HTP, were selected by two reviewers with good or very good agreement.
Data extraction Data relating to trial characteristics and effects of intervention on primary outcomes were extracted using a predesigned form. Risk of bias was assessed using Cochrane’s Risk of Bias tool v1.
Data synthesis 40 trials were included, 29 of which were tobacco industry affiliated. Methodological characteristics, such as registration, design, setting, comparator interventions, participants, outcomes and analyses, varied between trials, though there were few significant differences between industry-affiliated and independent trials. Of the 40 trials, 33 were judged to be at high risk of bias and 6 at unclear risk of bias. Trial findings were not significantly associated with either affiliation or risk of bias.
Conclusions The conduct and reporting of HTP interventional clinical trials were poor in many respects and limited to investigating effects of short-term exposure. These trials fall short of what is needed to determine whether HTPs are beneficial to public health, meaning they may not be a sound basis for tobacco control policy decisions.
- Tobacco industry
- Harm Reduction
- Non-cigarette tobacco products
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Previous research has shown industry-sponsored studies are more likely to have pro-industry results, potentially due to reduced quality and increased bias, yet the quality of interventional clinical trials on heated tobacco products (HTPs) and associations between findings on HTPs and affiliation or risk of bias have not been investigated.
WHAT THIS STUDY ADDS
Of the 40 identified interventional clinical trials assessing HTPs, 29 were industry affiliated and 11 were independent.
Many characteristics of these trials, such as short durations, confined settings and choice of comparators and participants, are not representative of real-world use and fail to adequately investigate whether HTPs reduce harm and are beneficial to public health.
Trial findings on the effect of HTPs relative to cigarettes were not significantly associated with trial affiliation or overall risk of bias.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
Existing intervention clinical trials on HTPs are largely inadequate in assessing the impact of HTPs on public health and may not, therefore, be reliable in tobacco control policy decision making.
Introduction
The harms of inhaling toxicants from combusted tobacco (ie, cigarettes) are well known.1 Heated tobacco products (HTPs) are designed to heat tobacco to relatively low temperatures. The purpose of this is to produce an inhalable nicotine aerosol which purportedly reduces the amounts of toxicants released and thus reduces health risks compared with cigarettes.2 The potential to reduce health risks is fundamental to HTP marketing3 and a contributing factor in their uptake and use by consumers.4–6 As HTP sales grow globally7 accurate assessment of their relative risks is essential. However, this assessment currently relies mostly on short-term laboratory research due to a lack of epidemiological studies.8 9
Previous reviews have highlighted the difficulties in interpreting the existing clinical evidence. The majority of clinical research into HTPs is conducted by the tobacco industry,8 9 which has a history of research manipulation.10 Tobacco industry studies largely show the potential health benefits of HTPs in smokers, while some independent studies have identified potentially harmful effects8 and found key industry studies do not comprehensively investigate all toxicants present.11 The association between a conflict of interest and industry-favourable findings has previously been observed in other tobacco and nicotine research.10 12 Poor or biased study design and reporting have been proposed as possible contributors to this phenomenon.10 Some methodological shortcomings have already been noted in HTP clinical research, such as short intervention durations, inconsistent reporting of data and potentially unethical practices, particularly in industry-affiliated studies.8 9 13–15 However, the quality of all HTP clinical trials has not yet been thoroughly examined.
Before consumers and policy makers make important decisions based on the results of these studies, it is crucial the quality of the evidence is assessed. Therefore, this review sought to critically appraise HTP interventional clinical trials by answering the following questions:
What are the methodological characteristics (ie, study details, design, interventions, participants, outcomes and analyses) and affiliations (ie, industry or independent) of interventional clinical trials on HTPs?
What is the risk of bias in these trials?
Are there differences in the methodological characteristics and risks of bias in industry-affiliated trials compared with trials with no industry affiliation?
What is the association between trial findings and: (a) trial risk of bias and (b) trial affiliation?
Methods
This systematic review followed recommendations set out by PRISMA.16 The protocol was registered on PROSPERO (CRD42021240676, https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021240676).
Search strategy and study selection
Web of Science (core collection and MEDLINE), Scopus, MedRxiv, ClinicalTrials.gov and the International Clinical Trials Registry Platform databases were searched on 28 April 2021. Searches were restricted to studies published from 2010 to exclude those on HTPs no longer marketed. Search terms included HTP terminology, brand names (‘IQOS’, ‘Ploom’, ‘Glo’) and clinical study terms (‘trial’, ‘participant’, ‘clinical’, ‘random*’). The online publication libraries of transnational HTP manufacturers (Philip Morris International, PMI; British American Tobacco, BAT; Japan Tobacco International, JTI; Imperial Brands, IB) and the reference lists of included literature were also searched. The full search strategy was reported in the protocol (https://www.crd.york.ac.uk/PROSPEROFILES/240676_STRATEGY_20210429.pdf). The searches were repeated to identify any relevant literature published between 28 April 2021 and 12 April 2022.
Trial publications were managed in Covidence. After duplicates were removed, title and abstract screening was piloted on 10% of the literature. Two reviewers (SB and AvdA) then independently screened all titles and abstracts, followed by full-text assessment against the eligibility criteria. Inter-rater agreement was measured using Cohen’s Kappa (k).
Inclusion criteria
Study design: Interventional clinical trials (studies in which human participants are prospectively assigned an intervention to evaluate its effects on health-related outcomes)17 of any design were included. Eligible studies did not need to be peer-reviewed or formally published.
Population: Adults (≥18 years).
Intervention: Studies were included if at least one arm was assigned a currently marketed HTP.
Comparison: Any comparator interventions.
Outcomes: Any outcomes.
Exclusion criteria
Studies published before 2010.
Studies that were not clinical trials.
Observational clinical studies.
Studies in which participants were not adults.
Studies in which an intervention was not a currently marketed brand of HTP.
Studies for which methodology and results data were not available, for example, ongoing studies.
Data extraction
Trial characteristic data were extracted into a predesigned form in Covidence by one reviewer (SB) and verified by a second reviewer (AvdA). The following data were extracted: study details (citation, country, trial registration date and ID, start and end dates, sponsor and affiliation); trial design (design, duration, comparators, setting); participant characteristics (eligibility criteria, age, sex, ethnicity, smoking history, comorbidities); intervention (type, cointerventions, mode of exposure); analysis (analysis population, unit of analysis, sample size calculation); outcomes (types, outcomes measured and reported, outcome matrices, time points measured) and results (participant flow, direction of effect in primary outcomes between HTP and cigarette groups at last follow-up). Two reviewers (SB and AvdA) independently coded trial affiliation. The full coding scheme is provided in online supplemental appendix 1.
Supplemental material
Last follow-up exhaled carbon monoxide means and SD were independently extracted by two reviewers (SB and AvdA). Where SD was not reported, it was calculated as per the Cochrane Handbook for Systematic Reviews of Interventions.18 Study authors were contacted to request missing data relevant to the meta-regression analysis.
Risk of bias assessment
Risk of bias was assessed using Cochrane’s Risk of Bias tool V.1.19 The assessment consists of six domains: random sequence generation and allocation concealment (selection bias); blinding of participants and personnel (performance bias); blinding of outcome assessment (detection bias); incomplete outcome data (attrition bias) and selective outcome reporting (reporting bias).
The evident differences between HTPs and comparator interventions means special considerations had to be made when assessing risk of bias. Unblinded trials were rated at low risk of performance bias if they were randomised and used an active comparator of similar intensity (ie, also contains tobacco/nicotine and all arms receive same cointerventions, if any). Unblinded trials were rated at low risk of detection bias if the primary outcome was objectively measured. Selection bias was rated high for all non-randomised trials.
The assessment was piloted on 20% of included trials (SB) and checked by an experienced assessor (JHB). Then, two reviewers (SB and AvdA) independently assessed risk of bias in all trials, resolving disagreements through discussion. The overall risk of bias for each trial was rated as ‘low’ when there was low risk of bias in all domains, ‘unclear’ when there was unclear risk of bias in ≥1 domains or ‘high’ when there was high risk of bias in≥1 domains. Risk of bias plots and graphs were generated using RobVis.20
Data synthesis and analysis
Trial characteristics data were summarised using descriptive statistics, distinguished by affiliation and tabulated where possible. Where comparisons involved two categorical variables, Fischer’s exact test was used to investigate associations between trial characteristics and affiliation. Due to inconsistent reporting and heterogeneity of available data, we could not conduct the meta-regression analyses per our protocol. Instead, we created an effect direction plot (as described in the Cochrane Handbook18 and by Boon and Thomson21) and used Fisher’s exact test to investigate associations between primary outcomes in each study and affiliation or risk of bias. We excluded studies with mixed effects for primary outcomes and/or were rated unclear risk of bias from these analyses.
Statistical analyses were conducted in Stata V.17. Significance level was 0.05.
Results
Included trials
A total of 987 and 214 records were identified through the first and second searches, respectively, of which 79 were included. There was good or very good agreement22 between reviewers for screening (first search: k=0.74, second search: k=0.81) and eligibility assessment (first search: k=0.64, second search: k=0.76). The 79 records related to 40 trials. Additional records pertinent to these trials (ie, registrations, protocols, reports and so on) were then collected, meaning 120 total records were included (figure 1). Key trial characteristics are provided in table 1 and full characteristics in online supplemental table 1. Two ‘actual use’ studies were identified. Typically used in pharmaceutical research, actual use studies investigate how a product is used under simulated real-world conditions.23 Although usually observational, these two studies met our definition of an interventional clinical trial and were, therefore, included.
PRISMA flow diagram for study selection. HTP, heated tobacco product.
Overview of included trials
Of the 40 trials, 11 (27.5%) had no known industry affiliation and 29 (72.5%) were industry affiliated. PMI conducted 16 trials, BAT conducted 7, JTI conducted 4 and JUUL conducted 1. The first and last authors of one study (Caponnetto, 2018)24 were funded by the Foundation for a Smoke Free World between 2018 and 2019, which was established with funding from PMI.25
Trial registration and reporting
Thirty-one trials (77.5%) were registered (figure 2). Only 12 (30%) were registered prior to enrolment of the first participant (ie, registered a priori). Most trials did not submit results for publication in a peer-reviewed journal (n=23, 57.5%) or post key outcome data on trial registries (n=26, 65%) within 12 months of trial completion (figure 3). Trial completion date was not reported in 12 (30%) trials; thus, timeframe for publishing results was unclear.
Number of trials that were registered on a clinical trial registry (‘Registered?’) and whether they were registered prior to enrolment of the first participant (‘Registered a priori?’). Size of bar indicates percentage of trials. Number within bar indicates number of trials.
Number of trials that reported results via peer-reviewed publications and posting on trial registrations within 12 months of trial completion. Size of bar indicates percentage of trials. Number within bar indicates number of trials.
There were no significant associations between affiliation and whether the trial was registered (p=0.08), whether it was registered a priori (p=0.70) or published results within 12 months of completion (p=0.07).
Trial design and setting
Thirty trials (75%; 20 industry-affiliated and 10 independent) were conducted in confined settings (ie, controlled environments, like clinics), 4 (11%; all industry-affiliated) in ambulatory settings (ie, uncontrolled environments, like participants’ homes) and 6 (15%; 5 industry-affiliated and 1 independent) in confined followed by ambulatory settings. Intervention duration ranged from single use up to 6 months. One BAT trial (ISRCTN81075760) was 12 months long, but at time of literature, collection results had only been reported for the first 6 months.
Thirty-four trials (85%; 26 industry and 8 independent) were randomised: 15 of parallel design, 18 crossover and 1 crossover followed by a case control study (table 1). The repeated measures study randomised the placement of interventions on participants’ skin, but all participants received all interventions and in the same order. Non-randomised designs included: a quasiexperimental trial, a non-randomised crossover and a study comprising two single-group assignment trials, one in which smokers used HTPs and one in which non-smokers used HTPs. In the two BAT actual use studies, products were allocated in random order within each group, but subject assignment to groups was not randomised.
There were no significant associations between affiliation and setting (confinement or ambulatory; p=0.25) or randomisation (p=0.32).
Interventions
The minimum number of intervention arms in any one trial was one and the maximum was eight. IQOS was the most common HTP intervention across both industry-affiliated (n=18) and independent trials (n=10). Excluding Caponnetto (2018), who used PMI’s IQOS, all industry-affiliated trials used the company’s own brand of HTP in at least one arm. Comparators included cigarettes, e-cigarettes, cessation, nicotine replacement therapy and non-smokers (table 1). Independent trials included an e-cigarette group significantly more often than industry-affiliated trials (p=0.0003). Only industry-affiliated trials included nicotine replacement therapies and cessation arms.
In most trials, participants used interventions ad libitum, regardless of confined or ambulatory setting. In seven confined trials, use was restricted (ie, puffing topography restricted). Three trials (9%) implemented both restricted and ad libitum use in confined settings and the mode of exposure was unclear in two (6%) trials. There was no significant association between mode of exposure and affiliation (p=0.27).
Participants
Four trials (10%; 3/29 industry-affiliated and 1/11 independent) failed to report the number of participants enrolled, randomised and/or completed. A total of 4098 participants were randomised (or enrolled in non-randomised trials) across the remaining 36 trials. A total of 3675 participants completed these trials, yielding an attrition rate of 10.3%: 10.5% across 26 industry-affiliated trials and 8.2% across 10 independent trials. Attrition was higher in ambulatory-only trials (average attrition=20%, n=3) than confinement-only trials (2.9%, n=28). Eighteen trials had withdrawals, 15 of which reported reasons for withdrawals and 3 did not.
Twenty-six (65%) trials reported baseline characteristics for the randomised/enrolled population, 8 (20%) reported them for the completed population, 5 (12.5%) reported them for analysis populations and 1 (2.5%) did not report any baseline characteristics. Based on available data, the mean age of participants was 40.1 years old and the ratio of male to female was 1.41:1 (n=4310 across 37 trials). In 35 trials, all participants were described as being in good health or without relevant morbidities. In one PMI trial, some participants had mild or moderate chronic obstructive pulmonary disease (COPD). In another PMI trial, all participants had chronic generalised periodontitis. Three trials did not report whether participants had any relevant morbidities.
Participants were smokers in all but two trials (NCT03889990/NCT03995329 and Dalrymple, 2022). Minimum eligible cigarette consumption across the trials ranged from ≥5 to ≥11 cigarettes per day and having smoked for ≥6 months to ≥10 years. One industry-affiliated and five independent trials did not define eligible smoking history.
Outcomes
A total of 214 different outcomes were measured across the 40 trials (online supplemental table 2). There was a wider variety of biomarkers of potential harm, but biomarkers of exposure were most measured (table 2). Number of outcomes measured in any one trial ranged from 1 to 71. The mean number of outcomes measured in industry-affiliated trials was 27 (mode=19, range=1–71), whereas for independent trials, it was 11 (mode=7, range=1–28). Seventeen trials (42.5%; 14/29 industry-affiliated and 3/11 independent; p=0.29) did not report results data for all outcomes measured.
Outcomes measured in heated tobacco product clinical trials
Analysis characteristics
A total of 275 trials (67.5%; 22/29 industry-affiliated and 5/11 independent; p=0.12) reported sample size calculations. The unit of analysis in 39 trials was individuals and areas of skin in 1 trial.26 The analysis populations used were: full analysis set (n=5, all industry-affiliated); full analysis set as exposed (n=3, all industry-affiliated); per-protocol population (n=5, all industry-affiliated); pharmacokinetic (PK) population (n=5, all industry-affiliated); per-protocol and PK populations (n=1, industry-affiliated); per-protocol and CEVal-compliant populations (n=1, industry-affiliated); not specified or unclear (n=20, 9 industry-affiliated and 11 independent). Population definitions are provided in online supplemental table 1.
Risk of bias
Thirty-four trials were judged to be at high risk of bias and for six trials risk of bias was judged to be unclear (online supplemental figure 1). Twenty-seven (93%) industry-affiliated trials were judged to be at high risk of bias and 2 (7%) unclear (figure 4A). Seven (64%) independent trials were judged to be at high risk of bias and 4 (36%) unclear (figure 4B) (significance not estimable as no low ratings). Judgement justifications are provided in online supplemental table 1.
(A) Risk of bias across industry-affiliated trials. (B) Risk of bias across independent trials. Size of bar indicates percentage of trials. Number within bar indicates number of trials. BPP, blinding of participants and personnel.
The 5 trials (3/11 independent and 2/29 industry-affiliated) judged to be at high risk of selection bias were due to these being non-randomised trials, meaning there was no random sequence generation or allocation concealment. There was no significant association between affiliation and rating for random sequence generation (p=0.07), but industry-affiliated trials had a significantly higher proportion of low ratings for allocation concealment than independent trials (p=0.0065). Selection bias could not be assessed for Dalrymple 2022 as the unit of randomisation was not individuals.
Risk of performance bias (blinding of participants and personnel) was judged to be high in 25 (86%) industry-affiliated and 2 (18%) independent trials (p=0.11). The numerous high ratings were commonly due to inability to conceal visually distinctive products and the control being non-active (cigarettes). As these factors are expected in HTP clinical research, we also determined overall risk of bias excluding this domain (figure 2; ‘Overall (exc. BPP)’). While this had no effect on overall risk of bias judgements across independent trials, 3 industry-affiliated trials went from high to unclear ratings, 10 went from high to low and 1 went from unclear to low. When excluding performance bias, there was evidence that industry-affiliated trials were judged to have low risk of overall bias significantly more often than independent trials (p=0.03).
Risk of detection bias (blinding of outcome assessment) was judged to be high in 3 (10%) industry-affiliated trials and 2 (18%) independent trials (p=0.5875). In all instances, this was due to some primary outcomes being subjectively measured in combination with either the trial being open-label (ie, no blinding) or a lack of information on blinding.
Risk of reporting bias (selective reporting) was high in 12 (41%) industry-affiliated and 4 (36%) independent trials (p=1). In all trials, this was because at least one outcome measured during the trial was not reported on at all in any trial literature.
Other biases were identified in two PMI trials due to all results data being grouped by participant product use (ie, ‘full analysis set as exposed’ analysis population), not randomisation.
Association between trial findings and affiliation or risk of bias
Table 3 (and online supplemental table 3) shows whether HTPs had a positive, mixed or negative effect on each trials’ primary outcomes compared with cigarettes at last follow-up. One independent study (NCT03889990/NCT03995329) had no cigarette arm and therefore direction of effect compared with the HTP was not estimable. Most industry-affiliated trials (59%) found HTPs had positive effects on primary outcomes compared with cigarettes, while most independent trials (60%) found they had mixed or negative effects. However, there was no convincing evidence that the proportion of effect directions was different between industry-affiliated and independent trials (p>0.05). We could not investigate associations between overall risk of bias and trial findings because no studies were rated low. Overall risk of bias judgements excluding performance bias were not significantly associated with trial findings (p=0.18).
Summary of direction of effect in primary outcomes at last follow-up between heated tobacco product and combustible cigarette arms
Despite attempting to adapt our methods, we were unable to perform the planned analysis. Nonetheless, the issues we encountered provide further insight into the quality of available data. First, there were few objectively measured outcomes which were measured in 10 or more trials (recommended minimum for meta-regression18) and measured in both industry-affiliated and independent trials. Data were also highly variable: last follow-up exhaled carbon monoxide (eCO) means ranged from 0.5 to 17.2 ppm across HTP arms and 0.8 to 25.6 ppm across cigarette arms. A possible solution to the issues of variability could have been to compare change in eCO from baseline to last follow-up, but few trials reported this. Moreover, the SD were relatively large compared with the means. This suggests the eCO data were positively skewed, as has been noted in other large population trials,27 28 yet most trials did not provide log-transformed eCO data.
Discussion
To our knowledge, this is the first study to critically assess the design and reporting of HTP interventional clinical trials and investigate associations between characteristics, affiliations and results. Despite worldwide use increasing,2 the number of clinical trials assessing HTPs remains low, especially those conducted independently of the tobacco industry, and most HTP trials were judged to be at high risk of bias.
In contrast with existing literature demonstrating industry sponsorship is associated with proindustry findings,12 29 we found no significant differences between findings from industry-affiliated and independent trials. Further, a 2017 Cochrane review found risk of bias did not differ between industry and independent studies, except for domains regarding blinding, which were more often rated low in industry studies. However, we found most industry-affiliated trials were at high risk of performance bias. When this was omitted, a significantly higher proportion of low overall risk of bias ratings were observed among industry-affiliated compared with independent trials. The differences between our findings and previous reviews’ findings may be due to the smaller sample size, most trials being limited laboratory-based studies of short-term exposure and using primary outcome data rather than overall conclusions of each trial to investigate associations. Additionally, selection bias could not be assessed in 1 of the 11 studies rated at low overall risk excluding performance bias. Full study reports were available for the other 10, which provided more information than can be presented in typical trial publications, like journal articles, thus reducing the chances of unclear judgements.
We noted numerous shortcomings in the design and quality of HTP trials. First, most trials were not registered a priori and did not publish results within 12 months, as recommended by the WHO and World Medical Association’s Declaration of Helsinki.30 31 Second, around half the trials did not report data for all prespecified outcomes. Selective reporting compromises the validity of trials, especially if significant outcome results are reported while non-significant results are omitted.32 It is disconcerting to find safety measures and biomarkers of potential harm particularly neglected given the health impact of HTPs remains uncertain.
Third, three independent and three BAT studies did not use a randomised controlled design and three PMI trials analysed data by exposure rather than random allocation, effectively derandomising the data. Lack of or compromised randomisation may reduce validity of results by creating an imbalance in subject characteristics (ie, possible confounding factors) between groups.33
Fourth, there were many characteristics which diminish the representativeness of the findings in real-world populations, including very short follow-up, which may not be long enough for adverse effects to manifest, and use of controlled confined settings. Many trials also used per-protocol or similar analysis populations, which exclude participants who deviated from the protocol or product assigned. In doing this, the trials can only estimate the effects of HTPs in ideal circumstances, that is, when smokers make a complete, or near-complete, switch from cigarettes. This may overestimate their true effects across real-world populations,34 in which consumers may use HTPs in conjunction with cigarettes or other products.
The choice of participants and products may also not be representative of real-world settings. Most trials included healthy participants, yet 12% of UK smokers report being in ‘bad’ or ‘very bad’ health35 and 15% of US smokers have COPD.36 Likewise, most trials did not include a ‘next best’ comparator based on options already available to smokers looking to reduce health risks, such as e-cigarettes, smokeless tobacco and nicotine replacement therapy. Notably, only five industry-affiliated trials included an e-cigarette arm, despite all the companies except PAX, manufacturing both HTPs and e-cigarettes.37 This could be to avoid directly comparing HTPs to a more established and popular competitor.
Although these short-term, confined trials can provide evidence on exposure to toxicants compared with cigarettes, they fall short of what is needed to determine whether HTPs reduce the risks of tobacco-related diseases and whether they are beneficial to public health in real-world settings. Furthermore, high risks of bias and notable weaknesses in trial conduct and reporting are concerning in regard to existing reviews by governments and health authorities, including in the USA,38 UK,39 Netherlands40 and Belgium,41 on which regulatory decision have been made. While methodological limitations were noted, most did not include systematic assessments of trial quality.
Although, to our knowledge, BAT’s actual use studies have not been reviewed by regulators, similar studies by PMI have.38 42 However, there is limited guidance on these studies and ethical approval can be complex to obtain.23 43 Indeed, ethical approval was obtained from BAT’s internal Human Research Committee in Jones 2020 and there was no mention of ethical approval in Gee (2018). In the absence of clear guidance, the design and reporting of actual use studies noticeably varies and raises concerns over their consistency and ethicality in tobacco research and regulation.
Strengths and limitations
This review included more trials than previous reviews,8 9 13 44 45 likely in part due to our less restrictive eligibility criteria. Following the guidance of the Cochrane Tobacco Addiction Group, we used Risk of Bias V.1 over the newer Risk of Bias V.2 tool because the latter requires an assessment for each outcome. This may have yielded different results, but it would have been impractical to do for all the outcomes we were interested in. Heterogenic data and inconsistent reporting meant the planned meta-regression analyses could not be conducted. Instead, we used direction of effect plots, but these do not consider statistical significance, the magnitude of effects or sample size differences between studies.
Conclusion
We found HTP interventional trials to be substandard in many aspects of their design and reporting, with most being at high risk of bias. Though our analyses detected few statistically significant differences between trials of different affiliation and risk of bias, this should only be interpreted as absence of evidence, not evidence of absence. Research in this area remains relatively sparse and results may change as further studies become available. The findings of this review highlight the inadequacy of existing clinical trial data in determining the health impacts of HTPs as used in real-world markets and thus calls into question their utility in regulatory decisions.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
Not applicable.
Acknowledgments
The authors would like to thank Professor Anna B Gilmore, Dr John Campbell and Dr Tom Hird for their guidance on the study protocol.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
X @Jamie Hartmann-Boyce@jhb19
Contributors The study was conceived by SB, who developed its design in conjunction with GMJT, JH-B and CM. SB and AVDA screened, coded and extracted data from study literature. SB and AVDA conducted the risk of bias assessments, checked by JH-B. SB drafted the manuscript, which was edited by all authors. SB is the guarantor and accepts full responsibility for the finished work and/or the conduct of the study, had access to the data and controlled the decision to publish.
Funding We acknowledge the support of Bloomberg Philanthropies’ Stopping Tobacco Organizations and Products funding (www.bloomberg.org).
Disclaimer The funders had no role in the study design, data collection and analysis, decision to publish or preparation of the manuscript.
Competing interests GMJT has previously received funding from Pfizer, who manufacture smoking cessation products. All other authors declare no competing interests.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.