With 10 years of tobacco documents research (TDR) completed, now is a reasonable time for those of us who conduct TDR to assess its legacy and its potential. In this issue of Tobacco Control, Stacy Carter thoughtfully depicts historical patterns in the conduct and reporting of TDR and how these patterns have evolved.¹ She proposes “a process for planning and evaluating TDR that positions the researcher as constructor” rather than merely as a conduit of information contained in the documents, and encourages researchers to be more conscious of the analytic traditions they bring to their searching and analysis strategies. From this platform, we explore the following: What has been the added value of documents research? How can this added value be sustained? And, by what standards should future work be assessed?

WHAT HAS BEEN THE ADDED VALUE OF DOCUMENTS RESEARCH?

TDR has helped us to better understand tobacco industry political and marketing strategies and research and design efforts, among others.² Carter identified 173 papers that used tobacco documents. Much of this work could certainly have been written without the documents, based instead on observed behaviour. But the documents have added important depth in three ways.

First, the documents confirm what we are able to observe. This confirmation makes it impossible for a Gray Robertson to deny an industry connection,³ or for scientists to deny they have accepted industry money,⁴ or for politicians to repudiate contact with the tobacco industry.⁵ They have also given us more insight into cigarette design.^6,7

Second, the documents have given us deeper insights into strategy. We can, for example, learn how the industry altered its framing of a political issue⁸ or how it segmented the population for targeting purposes^9–12 or worked to influence decision making in regulatory agencies.^13–15

Third, we are able to understand with more precision what happened when, and who was involved. In our work on labour unions and the industry,^8,16–18 for example, we can know which union leaders had industry contacts on which issues. We can also know who avoided those contacts and opposed the industry’s agendas.

CAN THIS ADDED VALUE BE SUSTAINED?

Thus, we believe that the added value of TDR is quite clear. But what’s next? Few would argue that the media is as interested in documents research now as it was when the first memos became available. Whereas a single memo could get a newspaper headline in the early years, now even a carefully researched paper is often greeted with indifference. Yes, the tobacco industry targeted various subpopulations; yes, they tried to influence legislation and elections (and sometimes succeeded); yes, they knew, yet lied about, their product being addictive and dangerous. These issues may still be alive legally, and have important implications for regulatory control of industry practices, but currently lack newsworthiness.

In academic research, however, gaining news attention is only one index of importance. We are also responsible for building a body of work that creates an accurate and thorough portrait of industry behaviour. This weight of evidence should have a long term impact on how the industry is viewed and regulated. So, what forms should that research take?

Carter has three findings that indicate how documents research has evolved—two we find encouraging, and one less so. She splits papers using tobacco documents into two groups: the A group that makes heavy use of documents; and the B group that only use a few documents. One finding we believe to be a positive step is that the number of purely descriptive A-papers is dropping while those focused on a more complex purpose or research question is increasing. If we are going to create a body of work that builds on itself, on public health or policy theory, or on other larger questions, we need to think about what important questions the documents can help us answer, rather than to create detailed papers that serve primarily to summarise documents.

Documents are a key data source for some research questions, such as those dealing with industry marketing efforts, political strategies, or cigarette research. A paper on any of these topics that lacks a documents component is likely to be incomplete. However, as with any good paper, we need the research question to drive the paper. And answering the research question will also likely require that we use sources in addition to the documents to confirm the information found there. For 10 years we have explored what the documents can tell us; maybe in the next 10 we need to focus more sharply on what it is we really want to know and then assess the usefulness of the documents in answering those questions.

The second encouraging finding in Carter’s work is that some of the B-papers use the documents to detail a “specific event or instance”, while relying primarily on other sources for the remainder of the paper. This use of documents as original source material indicates that their accessibility has encouraged others to begin using them, even if they are not the primary focus. Using documents as one of many sources, we suspect, will become more common, because the document collections are thin with regard to current industry behaviour or on issues related to sub-national governments.

Carter’s third finding, the one that concerns us, is that some B-papers make expansive statements about tobacco industry behaviour and then cite a document or two as evidence. Carter has identified a serious problem that should concern those doing TDR. If an author is going to make a claim that the tobacco industry targets immigrants or gay/lesbian people, it is better to cite papers that have done a thorough analysis of the documents and other sources. Such seemingly “off hand” uses of the documents may indicate a lack of understanding of what a long term effort it is to document patterns of industry behaviour.

BY WHAT STANDARDS?

Because documents research will continue, we should be concerned about the standards under which it is conducted. Here Carter opens an important conversation. By what standards should we conduct documents research? Can we set universal standards? If not, can we agree where we might disagree?

Carter suggests that, in fact, such a standard may be evolving. Her review of the A-papers by experienced documents researchers indicates that these papers are increasingly likely to state a purpose more complex than merely describing what is in the documents, to make use of other sources to confirm or contextualise what is found in the documents, and to describe their searching and analysis strategies.

Carter makes the excellent point that word limits in other types of research would not be an acceptable excuse for excluding information on methods or analysis. And we all want to use the scarce words we are allocated by editors to share our findings rather than our methods. But the reader should know something about how the authors conducted their work and reached their conclusions. While the process by which a research team assembles, sorts, and synthesises a large body of documents cannot be specified in the same level of detail as the process used by a team analysing a large number based dataset—the statistical protocols applied to the latter will always be more precise—readers should know enough about the process to trust the results presented.

Those of us who regularly search the documents understand the care that must be used in finding and synthesising information found in the documents. We have a strong interest in making sure that others understand that building such cases is not a casual or off-the-cuff effort. So, do we need to create a use and reporting standard?

While Carter suggests a reporting protocol, we are not sure that we are ready to establish one. Documents research is clearly continuing to evolve. Part of the difficulty in setting such a standard is that we do not know how documents will be used in the next decade. Based on the past decade, we can assume that people from multiple disciplines will use them, following multiple research traditions—positivist, interpretive, historiography. As Carter notes, all of these traditions are evident in current documents work. Thus, a single standard is perhaps not likely or desirable, if we think of the documents as a tool or a resource, rather than as a defining category of paper.

But those of us who use documents regularly, whether we agree on a universal reporting standard or not, certainly need to continue to document our methods more effectively and to use the peer review process to ask that others do so, too. We should embrace Carter’s call for greater transparency in our work—it will only make it better.