OBJECTIVE To describe and apply a process evaluation model (PEM) for patient education programs for pregnant smokers.
METHODS The preparation of a process evaluation plan required each program to define its essential “new” patient assessment and intervention procedures for each episode (visit) of patient–staff contact. Following specification of these core implementation procedures (p) by each patient education program, the PEM, developed by the Smoke-Free Families (SFF) National Program Office, was applied. The PEM consists of five steps: (1) definition of the eligible patient sample (a); (2) documentation of patient exposure to each procedure (b); (3) computation of procedure exposure rate (b/a = c); (4) specification of a practice performance standard for each procedure (d); (5) computation of an implementation index (c/d = e) for each procedure. The aggregate of all indexes (e) divided by the number of procedures (Pn) produced a program implementation index (PII = Σe/Pn).
PARTICIPANTS AND SETTINGS Data from four SFF studies that represent different settings were used to illustrate the application of the PEM.
RESULTS All four projects encountered moderate to significant difficulty in program implementation. As the number and complexity of procedures increased, the implementation index decreased. From initial procedures that included patient recruitment, delivery of the intervention components, and conducting patient follow ups, a variety of problems were encountered and lessons learned.
CONCLUSION This process evaluation provided specific insight about the difficulty of routine delivery of any new methods into diverse maternity care setting. The importance of pilot testing all procedures is emphasised. The application of the PEM to monitor program progress is recommended and revisions to improve program delivery are suggested.
- process evaluation model
- patient education program
- smoking cessation
Statistics from Altmetric.com
The national Smoke-Free Families (SFF) program, funded by the Robert Wood Johnson Foundation (RWJF), was initiated in 1994 to support the development and evaluation of innovative programs designed to help patients stop smoking during and after pregnancy.1 The SFF National Program Office (NPO) was established as part of this initiative at the University of Alabama at Birmingham, Department of Obstetrics and Gynecology. The NPO had four primary purposes: (1) to provide routine technical advice to the RWJF about issues related to this new initiative; (2) to manage the scientific review of SFF program proposals; (3) to provide technical assistance to grantees' patient education studies; and (4) to coordinate implementation of the studies.
One of the primary criteria used by the NPO in the scientific review of the proposals was an evaluation of the feasibility of routine delivery and replicability of the patient education intervention methods.1 In year 1 of the SFF program, all 11 grantees participated in several consensus development activities. They were asked: to apply a common set of definitions and questions to assess their patient samples at baseline and follow up; to measure changes in smoking status and tobacco exposure; and to assess the feasibility, patient acceptability, and costs associated with the delivery of the new patient education intervention methods. This paper focuses only on the issues of feasibility and replicability of delivery of intervention methods.
Program implementation success2-6 is an enduring methodological concern in all evaluations. Basch and colleagues7 have described the failure to implement a health education intervention as a “type III error”. Process evaluation data provide essential insight about what types of patient assessment and education methods can (and cannot) be routinely delivered for specific settings, behaviours, types of providers, and patients. This method should be used by a program as a quality control mechanism to assess staff performance of patient education clinical procedures. It also has a very practical function—to provide empirical information about salient structure and operational procedures within and across patient education programs in patient care settings.
Process evaluation data documenting the level of patient exposure to each patient education method are critical to making a conclusion about the efficacy, internal validity, costs and cost-effectiveness, and external validity of a patient education intervention.6-9A serious deficiency was reported in two comprehensive methodological reviews of 31 quasi-experimental and experimental evaluation studies of cessation methods for pregnant smokers.10 11 Although space limitations in a publication often restrict a study from providing a complete description of its patient education intervention, many evaluation studies lacked an adequate description and/or documentation of the delivery of experimental and control group methods. Only 12 of the 31 evaluation studies reviewed adequately described their intervention methods.11
Because of the salience of this issue for phase I and II of the RWJF SFF program and future patient education programs, this paper provides specific practice guidelines and examples to clinical staff about how to systematically evaluate program implementation. In this paper, we present: (1) a framework to describe patient assessment and education procedures for pregnant smokers; (2) a description of the patient flow analysis method; (3) a description of the SFF process evaluation model (PEM); (4) examples of the application of the PEM, using data from four different SFF supported patient education projects, including lessons learned; and (5) a discussion of issues for future process evaluations of patient education interventions.
DEFINITION OF PATIENT EDUCATION PROCEDURES FOR SFF
One of the first planning tasks all programs must perform is to describe what health education methods patients who smoke are supposed to receive as part of routine care and as part of an evaluation. Experimental and control group intervention procedures must be described, specifying the following: (1)what—the structure, procedures, and content of the patient education program used (for example, audiovisuals, written materials, and telephone counselling methods); (2)who—each member of the staff by degree/title who deliver the patient education procedures; (3)when—the frequency and periodicity of each patient contact by the type of method by type of provider; (4)how much—the time (estimated or observed) to deliver each patient education procedure at each contact; (5)setting—where and how the procedures are delivered (for example, group, one-to-one format, video or interactive computer).
Descriptions of these process variables are essential to: standardise study methods, develop staff training programs, recommend specific modifications of methods during pilot testing and at the end of an evaluation study, and replicate methods by future patient education programs and evaluation studies. Because “normal” patient education methods and staff behaviour will change owing to the presence of an evaluation, documentation of salient clinical procedures at each clinical visit is essential before and during the study.
Each SFF patient education program provided descriptions of their core clinical practice procedures. The example in table 1 for experimental group patient assessment and education procedures for four visits, including 400 pregnant smokers from four prenatal clinics from the Birmingham trial II,12 13 was provided as a prototype description for all sites. This example helped SFF project staff prepare descriptions of their procedures, personnel methods and materials, and to estimate time for each patient contact. As noted in table 1, this method defines what methods were supposed to be provided by whom at each visit. It is also an excellent method to document labour and non-labour costs for an intervention. Separate forms should be used to describe procedures for the experimental group and control group in an evaluation study.
PATIENT FLOW ANALYSIS
After describing the core clinical procedures to be provided to patients at each site, each program had to define how, when, and who would deliver the new methods. Patient flow analysis (PFA)14 is an excellent method to be applied when a clinical program is considering the introduction of new patient testing or counselling methods, such as smoking cessation. PFA documents patient exposure to specific services and staff at each clinic visit. The PFA method confirms patient time of arrival and exit at a clinic, and examines the patient time by type of visit, including the average time of visit, patient time receiving specific services, and proportion of time in contact and not in contact with clinical staff. It also helps managers and clinical staff to document who can deliver what kinds of services to whom, and the amount of time and cost for each patient with each type of provider. A sample of three to five patients at each site where the new methods are being considered for adoption is needed to document the specific time (in minutes) a patient spends with each provider and the type of services received. A detailed discussion of the PFA methodology,14 developed by the Centers for Disease Control and Prevention, is available from their clinic management unit.
ORGANISATIONAL DEVELOPMENT AND PFA
The introduction of any new set of procedures into a patient care system also requires policy, management, and practice support. PFA involves a collaborative process between program staff and managers who deliver and plan services.14 A PFA study should start with a careful examination of current patient care and education, policy, structure, process, and content to determine what will be delivered, to whom it will be delivered, when it be delivered, and where.14 Without participation of key staff from each clinical practice setting, it is very unlikely that a patient flow analysis will be conducted and/or its results useful or used by staff to plan how to integrate the new methods into care. This method helps to identify normal patterns to gain insight about possible adjustments to that pattern to maximise the opportunity for routine delivery of new procedures. SFF grantees were asked to review their patient care programs with managers and clinical staff to determine how best to integrate their new patient counselling methods for pregnant smokers or new parents.
SFF PROCESS EVALUATION MODEL
In a process assessment, the primary questions are as follows: (1) what procedures should a trained professional provide to a pregnant smoker or new parent at her first visit and follow up visits?; (2) what is excellent clinical practice for a specific behavioural problem and defined patient population?; (3) were the new procedures based on normative criteria (evidence) developed by a consensus of experts?; (4) did staff participate in the development of the implementation plan?; and (5) did staff perform the procedures?
Under the SFF PEM guidelines, each SFF grantee had the responsibility to define and implement a set of new clinical procedures. As discussed in section 1 of this report, the first step in the preparation of a process evaluation plan is to require each program to define its essential new patient assessment and intervention methods for each episode (visit) of patient–staff contact for an experimental group or control group patient.
Identification of the number of patients who were screened each week and recruitment of those who smoked—procedure 1—was the first task for all projects. Baseline data on patients screened, recruited, and patients who refused to participate, documents the daily, weekly, monthly, and annual patient census for each project by site. Among each eligible cohort of 100 patients who smoke (a), a number of patients (b) will be recruited at each site. This method produces information to compute an exposure rate (b/a = c) for each procedure. As noted in table 2, each SFF study had planned at the first visit a smoking status and psychosocial assessment of patients—procedures 2 and 3 (table 2). Patients in this example are scheduled to receive each of the next seven procedures at subsequent visits. Because this example includes experimental group patients only, procedures 7, 8, 9, and 10 would be performed. Procedures 4, 5, and 6 would not be performed for control group patients.
A practice performance standard (d), based on clinical practice guidelines, defines the expected level of patient exposure to a procedure. The NPO used 100% as an absolute practice standard (d) for each procedure. An implementation index (e) for each procedure is derived by dividing the exposure rate (c) by the practice performance standard (d). In this example, where the practice performance standard (d) is 100%, the implementation index (e) and exposure rate (c) are the same. A composite of all implementation indexes (Σe) provides a summary indicator of the successful delivery of a patient assessment and education program—program implementation index (PII).14 An overall PII > 0.90 would indicate an excellent level of implementation success.
APPLICATION OF THE SFF PEM
Illustrative data for the 10 clinical practice procedures for experimental group patients are also presented in table2.15 These hypothetical data indicate that the project needs to increase patient exposure to procedures 6, 7, 8, 9, and 10.
Each grantee had the responsibility to apply the SFF PEM to its sites to produce implementation data for its procedures. Implementation success, study progress, and problems, using monthly, quarterly, and annual data reports for each procedure, can be identified by this method. PEM data can be reported to a project coordinator by the 10th of each month for all eligible patients and all procedures. A staff training plan can be prepared to improve a specific exposure rate (c) or implementation index (e), when problems are documented—for example, when a rate or index falls ⩽ 90%.
SFF PROCESS EVALUATION CASE STUDIES
This section provides four case study examples of SFF grantee process evaluations.15-18 These studies were selected to illustrate four different types of interventions from four different geographic locations: (1) interactive video, (2) monetary–social support, (3) telephone counselling, and (4) tailored home videos. These four case studies reflect the challenges to recruitment and follow up assessments in diverse clinical settings. A description of the intervention procedures, application of the PEM, and lessons learned are presented for each case study. Because all of the case studies were evaluations of the efficacy of new procedures, a performance standard (d) of 100% was applied to all four cases for all procedures. Thus, the exposure rate (c) and implementation index (e) are equal in this discussion.
PATIENT ASSESSMENT AND COUNSELLING PROCEDURES: CASE STUDY 115
Pregnant smokers applying for the special supplemental nutrition program for women, infants, and children (WIC) were the target population in this study (table 3). The intervention consisted of three interactive video sessions and a final take home video. WIC participants came into the office on a monthly basis to pick up vouchers. Interventions were designed to correspond to these visits. On the first, second, and third monthly WIC visits after enrollment, experimental group subjects completed a brief questionnaire about smoking habits and cessation attempts during the previous month and viewed an interactive video. Each of the three, 7–10 minute videos started with a series of questions designed to assess current stage of readiness to change. Depending on each patient's stage, precontemplation, contemplation, and preparation, the computerised program chose one of three video pathways.
Each video pathway consisted of brief vignettes in which the patient was asked to “counsel” a pregnant woman who had just been told by her physician that she needed to quit smoking. Within this context, the patient was asked to answer questions the pregnant woman in the vignette had regarding the effect of smoking on the fetus, cessation methods, ways to cope with problem situations, and methods to manage relapse. A motivational counselling approach was the basis for the type and phrasing of the responses. On the subject's fourth visit, she completed a longer postintervention questionnaire, gave a saliva sample for cotinine assay, and was given a take home video on relapse prevention and creating a smoke free home. Each patient was given $10 vouchers to redeem at a local store upon completion of each of the four visits.
This was an exploratory study to evaluate: (1) how the WIC clients would respond to the new intervention; and (2) how the intervention could be implemented most effectively in the WIC setting. Enrollment in the program, intervention, implementation, and completion of data collection were done by research personnel. Maintaining contact over the four months of the study was the most difficult task. Transportation problems or sick children often caused women to miss WIC appointments or to send someone to pick up their voucher. Giving incentives helped to get patients to enroll in the program and to continue after a couple of visits. Making the monthly intervention brief (8–12 minutes) was also positive, because many patients voiced difficulty or unwillingness to spend much time doing the intervention beyond the time required for their WIC appointment. The time when the patients would normally be waiting to be called back to the voucher desk was used to complete the intervention. Most subjects thought the WIC clinic was an appropriate place for a smoking intervention, liked the interactive video method, but did not like seeing the same video on multiple visits if they failed to change stages from month to month. Almost all patients voiced a desire to cut down or quit smoking, but noted that stress was a big barrier to quitting. The WIC staff perceived significant difficulty if they had to routinely implement the intervention. Space was very limited and it was difficult to find a place where the patient could privately receive the intervention. Staff felt underpaid, very burdened by their current workload, and were unwilling to take on more responsibility. Many WIC staff were smokers, but did not always perceive that this behaviour might preclude them from counselling patients on smoking cessation.
PATIENT ASSESSMENT AND COUNSELLING PROCEDURES: CASE STUDY 216
In this study (table 4), experimental and control group participants were given verbal and written information on the importance of smoking cessation by WIC program staff, combined with a pregnancy/maternal-specific smoking cessation self help kit,A pregnant woman's guide to quit smoking.19 Participants in the experimental group were asked to designate a significant other supporter (SOS) partner. A brochure that suggested how to choose a positive supporter was given to each experimental group participant. A brochure that outlined specific ways for him/her to help his/her “partner” was mailed to the SOS partner. A written instrument was also mailed to the SOS partner with a stamped, self addressed envelope to return the consent form and the instrument to the main office. Experimental group participants were encouraged to select an SOS partner who would be the most likely person to offer positive support. All SOS program participants were requested to return to the WIC site a minimum of two additional times over the course of the maximum 10 month program intervention period. During the baseline and two follow up assessments, saliva was collected to be analysed for thiocyanate and cotinine, and written instruments were administered. Additionally, each month, SOS staff contacted each participant by telephone to collect her self report smoking status assessment based on the question, “Have you smoked a cigarette, even a puff, in the last seven days?”. If the participant stated “No”, an appointment was made for her at the WIC site to collect a saliva specimen. If the specimen confirmed her quit status, the experimental group participant and her designated SOS partner received incentive vouchers in the mail.
A minimum financial incentive was given to all patients to encourage recruitment and follow up participation. It helped to overcome the net effect of barriers to participation—transportation, distance, and time. The participation stipend contributed to the implementation index over time.
Provision of an incentive/stipend directed to each local WIC program enabled both access to WIC participants and continued WIC staff participation. Use of local department store vouchers for participants seems to be a viable and effective alternative to cash or other rewards. The vouchers were stamped, “not for alcohol or tobacco products”. Consistent with other research, an accessible point of contact through a “1-800” SOS program main office telephone line proved invaluable for participants to access office staff to answer their questions and allowed participants to return calls prompted by staff messages. This population was very transient in nature. It was not at all unusual to lose then regain contact with participants. It took an average of three telephone calls to each patient each month (average of three intervention months between baseline and follow up) to collect the monthly self report smoking status assessment.
A variety of methods was used to track patients over the intervention time line: (1) WIC program database; (2) friend/family telephone number collected at recruitment; (3) SOS partner telephone number collected at recruitment; and (4) local telephone carrier information services. To maintain smooth delivery of the educational intervention, WIC staff completed the primary training and then received a secondary training approximately six months later to support the routine delivery of the educational intervention.
PATIENT ASSESSMENT AND COUNSELLING PROCEDURES: CASE STUDY 317
After patients had undergone a telephone screening for inclusion in this study (table 5), each was asked to set a quit date within the next two weeks and mailed treatment materials. Women in the control group (usual care) received the video intervention for pregnant smokers (VIPS) self help quit calendar and cessation tip guide. Women in the experimental group (usual care plus video) received the calendar and tip guide plus the video program. All follow ups were conducted by telephone. Major patient assessments were conducted 2–3 days after the quit date, 4–5 weeks after the quit date, and one month postpartum. Measures of abstinence, negative affect, coping, stress, and self efficacy were obtained by phone interviews only. No counselling was provided during any of the phone follow up visits.
The most significant difficulties encountered in this study were patient recruitment and patient retention. Only 50 of the 82 women completed all follow up assessments. While there is appeal for using videos as a minimal intervention, the lack of personal contact may have contributed to poor compliance. Moreover, to take full advantage of this medium a woman must make a concerted effort to keep a viewing schedule. It might be more convenient to keep a “video viewing” appointment in one's home, in contrast to keeping a clinic appointment with a cessation counsellor. However, the freedom to watch the videos at any time also creates an opportunity for significant distraction as the activities of daily living inhibit video appointments. This lack of commitment reduced compliance. Patient commitment was an issue in this study. For example, even though all women agreed to set a quit date within two weeks of study entry, only 21% of the experimental (video) group and 33% of the control group endorsed “complete” abstinence (never smoke again) as their treatment goal. Commitment to total abstinence, as described here, has been shown to be a predictor of success in other efficacy trials.20
The data showed that this population was also highly nicotine dependent: ⩾ 50% smoked in the first 30 minutes of the day. In addition, about 50% of the women either met the criteria of theDiagnostic and Statistical Manual of Mental Disorders (fourth edition) for major depressive disorder on the PrimeMD,21 or had CES-D22 scores > 16. Taken together, these factors—poor commitment, high nicotine dependence, and affective disturbance—may have contributed significantly to the overall poor compliance with the patient education procedures and low cessation rates.
PATIENT ASSESSMENT AND COUNSELLING PROCEDURES: CASE STUDY 418
In the fourth study (table 6), participants were assessed on smoking and psychosocial variables at their first (baseline), second, third, fourth, and end-of-pregnancy (28–34 weeks) prenatal visits. All assessments were conducted by a nurse research assistant who administered the 5–10 minute questionnaire out loud at the beginning of each visit. Urine samples were collected for cotinine analysis at the baseline and end-of-pregnancy visits. All patients received brief smoking cessation counselling (typically lasting three minutes) and stage appropriate printed materials from their obstetrician or nurse midwife at the first, second, and third prenatal visits. Patients randomised into the experimental group, who had moderate or high intentions of quitting smoking during their pregnancy and a home telephone, were offered the opportunity to receive support over the telephone from a specially trained female ex-smoker. The proactive telephone support typically occurred on a weekly basis. Each call averaged 10 minutes in length, and women could request to stop getting the calls at any time. After each contact, the support person wrote down on a log sheet the length of the call and the patient's current smoking status. Quality control checks were made by the project director to verify contacts.
No difficulty was experienced in administering the baseline and end-of-pregnancy assessments, collecting the urine samples, or conducting the practice based intervention components, because these procedures were completed by a nurse research assistant in the prenatal practice. This full time, dedicated staff person screened, recruited, and assessed our patients and put a protocol prompt sheet on each participant's chart to guide the obstetrician or nurse midwife in the appropriate brief smoking cessation counselling at each visit. The providers staffing the clinic liked this process, especially because the combined assessment and intervention time was kept to about 10 minutes to avoid disrupting the patient flow in the busy medical practice. The nurse research assistant also helped out with obstetric nursing duties when not conducting research activities. These arrangements combined to produce 90–100% adherence to assessments and practice based interventions.
An attempt was made to recruit all pregnant women smokers attending the practice for their first prenatal visit into the study to test the impact of the proactive telephone peer support intervention—an effectiveness trial. Therefore, all women smokers—regardless of their intentions to stop smoking during their pregnancy or their access to a home telephone—were recruited. However, women randomised to the experimental group were only offered the telephone peer support if they had moderate or high intentions of stopping smoking during pregnancy and a home telephone. Because nine experimental subjects had low intentions of quitting, and nine had no home telephone, only 77% of the women in the experimental group were offered the telephone peer support. Of the 59 experimental subjects offered the telephone support, 44 women (75%) accepted the offer and 71% were reached for at least one support contact. Although a slight reduction in the percentage of women who received four or more support calls (62%) was observed, generally once women accepted the support, contacting them was usually successful. This is an important observation, as most of the sample were women of lower income who traditionally have multiple barriers to the receipt of external support. The proactive telephone intervention was successful in retaining those women who initially accepted the offer.
Because the study was conducted as a test of the feasibility and effectiveness of a telephone peer support intervention, there was not sufficient power to test the efficacy of the intervention. The process data gave a good indication of the acceptance of the telephone peer support offering in a real world situation. In retrospect, it may have been wiser to have conducted the research first as an efficacy trial before attempting to simulate true practice conditions. This was the greatest lesson learned.
When a cross project comparison is performed of the four examples, several conclusions can be made. The initial psychosocial and smoking status assessment (with the exception of case study 3—saliva collection = 0.62) were successfully completed. All projects, however, encountered difficulty in implementing their patient education procedures: case study 1 = 0.83, case study 2 = 0.81, case study 3 = 0.50, and case study 4 = 0.77 (patient education procedures only). As the number of patient contacts increased, the exposure and feasibility level decreased. The efficacy and potential behavioural impact was attenuated in each study by the reduced exposure to the patient education procedures. Lost to follow up presented (one exception = 0.91) an additional problem. Overall, for these studies and future evaluations, unless all procedures can be performed at > 0.90 or better, the internal validity of results becomes increasingly diminished or is totally compromised. Without process, impact will not occur.
As documented in the examples, the central value of the PEM is that it can provide data for monthly progress reviews that can be used to identify specific implementation problems. Focus group methods with patients or staff training can also be performed to eliminate or significantly reduce barriers to routine delivery and/or use by patients. In case study 3, for example, six videos for patients were created. The process data confirmed almost no use of the videos beyond video 3 or video 4. These data, combined with patient feedback, require a reduction of the number of videos. A somewhat analogous situation was observed in case study 4 in which telephone counselling (four calls) was used. The data suggested that a reduction in the number of calls from four to three would be advisable. Each of these illustrations provides practical, empirically based data that should be combined with other qualitative information to modify a program.
The PEM is also the primary method used to prepare a cost analysis of new and existing patient education programs. The routine application of the PEM documented the degree to which the clinical staff of the four examples had implemented all procedures as planned. The PEM provided empirical insight about the feasibility of routine delivery and replicability of procedures at comparable settings. Future studies should apply the PEM in program planning and evaluation.
If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.