Identifying best modelling practices for tobacco control policy simulations: a systematic review and a novel quality assessment framework

Background Policy simulation models (PSMs) have been used extensively to shape health policies before real-world implementation and evaluate post-implementation impact. This systematic review aimed to examine best practices, identify common pitfalls in tobacco control PSMs and propose a modelling quality assessment framework. Methods We searched five databases to identify eligible publications from July 2013 to August 2019. We additionally included papers from Feirman et al for studies before July 2013. Tobacco control PSMs that project tobacco use and tobacco-related outcomes from smoking policies were included. We extracted model inputs, structure and outputs data for models used in two or more included papers. Using our proposed quality assessment framework, we scored these models on population representativeness, policy effectiveness evidence, simulated smoking histories, included smoking-related diseases, exposure-outcome lag time, transparency, sensitivity analysis, validation and equity. Findings We found 146 eligible papers and 25 distinct models. Most models used population data from public or administrative registries, and all performed sensitivity analysis. However, smoking behaviour was commonly modelled into crude categories of smoking status. Eight models only presented overall changes in mortality rather than explicitly considering smoking-related diseases. Only four models reported impacts on health inequalities, and none offered the source code. Overall, the higher scored models achieved higher citation rates. Conclusions While fragments of good practices were widespread across the reviewed PSMs, only a few included a ‘critical mass’ of the good practices specified in our quality assessment framework. This framework might, therefore, potentially serve as a benchmark and support sharing of good modelling practices.


7
Text S3. Summary of included models 14 Text S4. List of papers not included in this systematic review 20  Figure S1. Model score and number of peer-reviewed publications linked to the model BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Text S2. Potential Good Modelling Practices.
We examined the modelling approaches by a) model inputs (hierarchy of evidence, population representativeness), b) model structure (exposure granularity, disease epidemiology, documentation), and c) model outputs (reporting standards, uncertainty and sensitivity analysis, model validation) to identify method strengths and weaknesses.
BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

SimSmoke
SimSmoke is a first-order Markov model to estimate the smoking prevalence changes and smoking-attributable deaths of various tobacco control policies. SimSmoke relies on four sub-modules -population size, smoking prevalence, smoking-attributable deaths, and policy modules. Risk factors included categorical smoking status and the year since quitting. Model outcomes focus on mortality and smoking prevalence. SimSmoke was calibrated, and sensitivity analysis was performed. Readers are provided with the model documentation. The model was reported with external validation. However, there were no simulated diseases mentioned in the model. SimSmoke was also used to model smoking behaviour by dual users (SLT and cigarettes or snus and cigarettes).

Abridged SimSmoke
Abridged SimSmoke is a model that uses a single year to project policy short-term (5 years), mid-term (15 years), and long-term (40 years) effects on smoking prevalence and smoking-attributable deaths. Slightly different from the four modules in SimSmoke, Abridged SimSmoke utilises three components population size, smoking prevalence and policy modules in the approach. In this model, populations are stratified with an unemployed status.

Extended cost-effectiveness analysis (ECEA) tobacco tax model
The extended cost-effectiveness analysis (ECEA) tobacco tax model is a cost-effectiveness model in estimating the impact of tobacco taxation. It was adapted from the Asian Development Bank's framework. The population groups were stratified by income quintile. It included diseases such as COPD, CVD, stroke, lung cancer, bladder cancer and BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) Moreover, it was tested with one-way sensitivity analysis and validated. The model technical document is available for readers. Nevertheless, the majority of the studies using this model focused on male-only.

IMPACT
IMPACT is a cell-based model to estimate CHD mortality changes under different policy scenarios. Risk factors included blood pressure, cholesterol, diabetes, fruit and vegetable, smoking (never smoker, long-term ex-smoker, recent ex-smoker, current smoker), salt intake, saturated fat intake, BMI and physical activities. Model simulated diseases include CHD and type 2 Diabetes. In the IMPACT model, population characteristics include age, gender and socioeconomics classes (indicated by QIMD). The model projects outcomes on equity, CHD mortality, smoking prevalence and life-years gained. Moreover, the resulting uncertainty was reported. Probabilistic sensitivity analysis (PSA) using the Monte Carlo approach was applied as the sensitivity analysis, and the model was externally validated. Moreover, the model documentation is available to readers.

European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD)
The European study on Quantifying Utility of Investment in Protection from Tobacco model (EQUIPTMOD) is constructed as a Markov state transition model. It models smoking cessation on four diseases: stroke, lung cancer, coronary heart disease and COPD. It provides economic estimates on intervention cost, return on investment (ROI), incremental cost-effectiveness ratio (ICER) and quality-adjusted life-year (QALY). Both univariable sensitivity analysis and PSA were performed. Technical document for all countries is available on the study website. However, there was no model validity mentioned in the papers.

Benefits of Smoking Cessation on Outcomes (BENESCO) model
Benefits of Smoking Cessation on Outcomes (BENESCO) model is a discrete-time Markov model that estimates the cost-effectiveness of a single smoking cessation attempt. Smokers were modelled by quit smoking duration, including smoker, recent quitter and long-term quitter. COPD, CHD, stroke and lung cancer were included in the model. Results on mortality, morbidity, cost and QALY were generated. In addition, univariable sensitivity analysis and PSA was performed on this model. It was calibrated. However, there is no documentation provided. In addition, funding was provided by Pfizer. BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s) One-way and PSA were performed for this model. Scenario testing and face validation were applied for this model.

DYNAMO-HIA model
The DYNAMO-HIA is a software applying a discrete-time, Markov-type multistate model. The model combines a microsimulation to simulate the risk factor exposure development and projecting the health impact over time with a macrosimulation. Moreover, three modules -population, disease, risk factors were included; eight health risk factors were included -BMI, alcohol, smoking, second-hand smoking, salt intake, physical activities, obesity. The model simulates nine smoking-related diseases: ischemic heart diseases (IHD), diabetes, COPD, stroke, lung, breast, colorectal, oral, and oesophageal cancer. The model estimates the chances of morbidity and healthy life years (HLY).
The model validity checked was mention for this model.

Johansson model
Johansson model is a Markov-cycle tree model. It simulates smoking cessation on COPD, cardiovascular disease (stroke and CHD) and cancers to estimate QALY and cost impact. Sensitivity analysis was performed using multivariable analysis and PSA. Model external validation was mentioned. Moreover, the model non-technical document is available.

Prevention Impacts Simulation Model (PRISM)
Prevention Impacts Simulation Model (PRISM) is an interactive system dynamics model for cardiovascular disease prediction. Users could interact with the model parameters using the user interface. It was designed to estimate policy impact on mortality, morbidity, healthcare cost, productivity and result uncertainty. A series of risk factors were included: blood pressure, cholesterol, second-hand smoking, obesity, psychological distress, fruit and vegetable, smoking (never smoker, long-term ex-smoker, recent ex-smoker, current smoker), blood glucose categories, periodontal disease, sleep apnoea, small particulate air pollution, and inadequate use of aspirin for primary prevention.
The model was externally validated, and the sensitivity analysis was checked with PSA. However, it was only applied to the US setting.

Barnett model
Barnett model is a Markov model that used for smoking cessation trial cost-effectiveness. Treatment effectiveness is extracted from the trial. It predicts the trial lifetime effect on cost, mortality and QALYs. The result range is provided.
This model was tested with a one-way sensitivity method. Its technical appendix is provided, but the code is not opensource. The model was calibrated; however, there was no mentioning of the model validation and no specific modelling of diseases mentioned for this model.

Cantor model
The model designed by Cantor et al. is a two-structured decision-analytic model to assess the cost-effectiveness of smoking cessation interventions over a lifetime. The first model evaluates cost per successful quit while the second one estimates life expectancy and quality-adjusted life expectancy. This model includes a lifetime horizon to capture the smoking intervention for long-term benefit-however, the model only simulated interventions in the United States.
One-way and two-way sensitivity analysis were used. The model validation is not mentioned, and there is no additional model documentation provided.

Chevreul model
Chevreul model is a Markov state-transition model that is used to predict cost-effectiveness analysis of smoking policies on the French population. The model simulates the natural history of smokers until death. It only modelled smokers diagnosed with either lung cancer, COPD or CVD, such as stroke or coronary artery disease and death.
Diseases include COPD, CVD and lung cancer. Moreover, health outcomes and ICER are provided by the model. The model used sensitivity tests and was cross-validated. The model documentation is available.

Parrott model
Parrott model is used in evaluating the cost-effectiveness of clinical trials over a lifetime. The policy effectiveness was extracted from a randomised controlled trial, and other data inputs were either from the trial or national representative surveys. Diseases including COPD, CHD, stroke, lung cancer, asthma, pregnancy-related (placental abruption, ectopic pregnancy, pre-eclampsia, placenta previa and miscarriage infant morbidities: low infant birth weight, stillbirth, premature birth) were modelled. The model estimates trial outcomes on cost and QALY with a result uncertainty BMJ Publishing Group Limited (BMJ) disclaims all liability and responsibility arising from any reliance Supplemental material placed on this supplemental material which has been supplied by the author(s)

Population Health Impact Model (PHIM)
Population Health Impact Model (PHIM) is a tobacco industry funded model by Philip Morris International. This model evaluates the health impact of a candidate modified risk tobacco product (cMRTP). It projects cMRTP uptake and mortality rate changes under alternative scenarios. cMRTP users and dual users were counted as the smoking status. In addition, smoking-related attributable deaths from lung cancer, ischemic heart disease, stroke and chronic obstructive pulmonary disease were considered. This model was tested with sensitivity analysis and validated. PHIM model comprises two modules -a population module that generates distributions of smoking histories for each scenario at the end of the period being studied and an epidemiologic risk module to estimate smoking-related attributable deaths.

Tobacco Town
This is an agent-based model.

Coronary Heart Disease (CHD) Policy Model
Coronary Heart Disease (CHD) Policy Model is a state-transition Markov model that predicts policy impact on CHD incidence, prevalence, mortality and costs. This model includes three sub-models: demographic-epidemiological, bridge and disease-history. Six risk factors linking with CHD and stroke were simulated in this model. Moreover, this model was calibrated, and sensitivity analysis was performed.

Lung Cancer Policy Model (LCPM)
The Lung Cancer Policy Model is a state-transition microsimulation that models lung cancer development, screening and treatment at the individual patient level. Detailed patient smoking histories were counted in this model. This    Specify key study characteristics (e.g., study design, risk of bias) used to order the studies in the text and any tables or graphs, clearly referencing the studies included