IT-Enabled Systems Engineering Approach to Monitoring and Reducing ADEs

Objectives: To develop and pilot-test a Web-based implementation of a team resource management (TRM) intervention to improve medication safety in primary care.

Study Design:

Randomized controlled trial.

Methods:

Eight practices were randomized to either the Web-based TRM or usual practice (4 practices in each group). Primary outcome was adverse drug events (ADEs) in older adults, ascertained using a trigger tool chart review at two 12-month periods (before and after the intervention). The prospective TRM approach, designed to inculcate ownership and empowerment, facilitates systematic appraisal of risk and error reduction. This approach uses the highly adaptable and transferable Safety Enhancement and Monitoring Instrument that is Patient Centered.

Results:

The rate of ADEs decreased from 25.8 to 18.3 per 100 patients per year in the intervention group. The rate was virtually unchanged in the control group (24.3 vs 24.8). In an analysis of covariance at the practice level, being in the intervention group was associated with a lower rate of ADEs. The interaction between time (preintervention vs postintervention) and group (intervention vs control) was not signifi cant (P = .104) but showed a trend toward a decrease in the intervention group compared with the control group over time.

Conclusions:

The Web-based TRM intervention proved feasible and demonstrated potential for effectiveness in various ambulatory settings. This pilot study was limited by small size and short follow-up period. Future studies should test the intervention on a larger scale over a longer period of time and should explore methods for overcoming common barriers to change.

(Am J Manag Care. 2012;18(3):169-175)This randomized controlled trial pilot-tested a Web-based implementation of a self-empowering team resource management (TRM) intervention aimed at improving medication safety in primary care settings.

•  The intervention sites showed a significant reduction in adverse drug events.

•  Interaction between time (pre- vs postintervention) and the 2 arms of the study was not statistically significant.

•  This study suggests that the Web-based TRM has the potential to improve medication safety in busy primary care offices.

Medication use is recognized to be a high-risk activity across all settings. A 2007 Institute of Medicine (IOM) report on this subject acknowledges that the rates and impact of medication errors are huge but are poorly understood.¹ In ambulatory settings, medication errors and adverse drug events (ADEs) are important safety issues. Gurwitz and colleagues have estimated (by extrapolation) that Medicare enrollees alone suffer approximately 500,000 preventable ADEs per year.²

Lack of awareness of the type, incidence, and consequences of errors in any setting is one of the most important barriers to reducing these errors as well as improving safety and quality of care. The most commonly used method for estimating vulnerabilities in healthcare is to retrospectively collect and count errors through voluntary reporting systems (often referred to as incident reports). These are fraught with difficulty due to various issues, including underreporting; according to IOM’s 2000

report, only 5% of known errors are typically reported.3 Error reporting often does not promote understanding of the organizational structure and processes of care. Instead it tends to be associated with blame and shame, and frequently results in antagonism between team members that undermines mutual respect, trust, and cooperation. Bates and colleagues have described diffi culties involved in defi ning and quantifying errors; they report that even direct observational studies, which are highly labor intensive, often miss errors.⁴

An alternative approach that is prospective rather than retrospective, and encourages involvement of all team members in identifying and prioritizing safety and quality problems, invokes failure modes and effects analysis. This type of analysis has been widely used in other high-risk industries and has been advocated by the IOM as a means of analyzing a system to identify its weaknesses (failure modes) and possible consequences of failure (effects), and to prioritize areas for improvement.3 We adapted and tailored this methodology to allow for the levels of resources and expertise available in ambulatory settings and developed an instrument that has been well received by staff in a variety of settings. The details of the concepts, rationale, and processes behind this highly transferable instrument, termed Safety Enhancement and Monitoring Instrument that is Patient Centered (SEMI-P), are described elsewhere.^5-7

The objectives of this study were to develop and pilot-test an information technology (IT)—based team resource management (TRM) system, based on SEMI-P, aimed at improving medication safety in primary care. We examined (1) the ability of this intervention to reduce selected ADEs among geriatric patients, (2) the ability of this intervention to improve monitoring of geriatric patients taking selected chronic medications, and (3) how offi ce staff used and applied this IT-based TRM tool for improving geriatric medication safety.

Study Design

This was a randomized controlled trial of a Web-based TRM intervention to reduce ADEs in primary care. Randomization was at the site level; 4 sites were assigned to the intervention and 4 to a control state (usual practice). In all 8 sites, ADEs were ascertained using a previously published trigger tool methodology.^2,8

All practices in the current IT-facilitated study were part of the Upstate New York Practice Based Research Network and had electronic medical records in place for at least 12 months prior to the start of the study. Both groups contained a variety of practice types including safety net practices. Urban, suburban, and rural practices of various sizes and with various ownership structures were represented. All staff at the above sites—including physicians, physician extenders, nurses, medical assistants, administrative staff (secretarial and management), and all others (eg, dieticians and social workers if present)—were invited to participate in surveys and team discussions.

The project addressed the IOM priority areas of frailty associated with old age and medication management. The study protocol was approved by the Social and Behavioral Sciences Institutional Review Board of the State University of New York at Buffalo.

METHODS

The intervention involved implementation of the Webbased TRM system in the 4 practices that were randomly assigned to this group. The system uses a cyclical safety improvement process facilitated by use of 2 anonymous online staff surveys. The fi rst survey instrument is SEMI-P, which is a failure modes and effects analysis tool designed for the ambulatory setting that focuses on medication management processes.^5-7 The second instrument is the safety attitudes questionnaire— ambulatory version (SAQ-A), which provides measures of the safety climate.^9,10 Among the published safety climate surveys available at the time of this study, the SAQ was determined by Colla and colleagues to have the best psychometric properties.¹¹ For each survey, online instructional clips and automated analysis with visual presentation of results were developed and implemented as part of the IT system.

Figure 1

is a screen shot of the first of 12 pages of the online SEMI-P survey. Each member of every intervention practice (including all job categories) was invited to complete this survey anonymously as part of a staff meeting. The same procedure was followed for the SAQ-A.

Figure 2

Figure 3

At each site, after administration of each survey, the practice teams immediately regrouped and reviewed the results and commenced discussion. is a screen shot showing the compiled results of the SEMI-P, generated by the system in real time. shows example results from the SAQA in a visual format that was developed especially for this project to highlight strengths and weaknesses in the safety culture.

These steps were followed by a series of staff meetings, held during times when no patient care was being delivered (generally at lunchtime), which all available staff attended. At these meetings, the survey results were reviewed and discussed, leading to consensus-based prioritization of medication safety issues. This consensus was also aided by visualized displays of individual (anonymous) team members’ opinions. Examples of prioritized areas included poor patient education about medications, high no-show rate, poor medication tracking, and poor coordination/teamwork with respect to handling of medication refi ll requests.

eAppendix

In subsequent staff meetings, teams worked together to address the chosen priorities by developing feasible system changes to improve medication safety. Examples include consistent use of patient education materials for high-risk medications, inclusion of diagnosis on prescriptions, patient reminders regarding follow-up, patient-carried medication lists, redesign of medication refi ll work fl ow, better training and follow-up for new personnel, and better employee performance feedback. The figures in the (available at www.ajmc.com) illustrate the visual aid to consensus forming and the overall view of the whole methodology.

The initiatives tool within the software was used by staff to defi ne their goals and objectives for safety improvement, identify and assign specific work steps to individual team members, track progress, coordinate meetings, and remind staff of their commitments. The indicators tool was used to define specific measurable outcomes related to each initiative and to graphically track these over time in order to determine whether the stated objectives were being met.

Outcome Ascertainment

The primary outcome at the 4 intervention and 4 control sites was the rate of ADEs (measured using a trigger tool methodology). A secondary outcome was compliance with Healthcare Effectiveness Data and Information Set (HEDIS) guidelines for laboratory monitoring for patients who were prescribed certain medications chronically (meaning that they were prescribed the medication for 6 or more months out of a 12-month period). Both of these outcomes were for older adults (aged >65 years) since these patients are known to be at higher risk of adverse events.

A Web-based data capture tool was developed and implemented as part of the IT system for both the trigger tool and the HEDIS measure.

These 2 outcomes were ascertained for a baseline period defi ned as the 12 months before the start of the intervention and an end point period defi ned as the 12 months after the start of the intervention. For each period (baseline and end point), research assistants reviewed an independent sample of 100 charts of patients aged >65 years at each site. The HEDIS laboratory monitoring measurement was completed by these research assistants. For the trigger tool, the research assistants conducted the fi rst of 2 steps, known as the screening step. Charts identifi ed in this fi rst step as having triggers underwent secondary review (the review step) by a physician or pharmacist who reviewed each trigger to determine whether an adverse drug event occurred, and if so, its severity and preventability.

Analysis

Rates of ADEs were calculated for each practice, normalized per 100 patient-years at both the preintervention and postintervention time periods. Chi-square and t tests were used (where applicable) to determine baseline differences in patient characteristics between the intervention and comparison groups. Variables with significant baseline differences were included as factors in an analysis of covariance (ANCOVA). This analysis was at the site level. All analyses were performed using IBM SPSS version 19.0 (IBM Corporation, Armonk, NY).

RESULTS

Patient Characteristics

Table 1

The demographic characteristics of the patients included in the outcome ascertainment are shown in . The vast majority had cardiovascular disease; about a third had diabetes mellitus. The average patient had 5 comorbid conditions and was on 7 medications.

Rates of Adverse Drug Events

Rates of ADEs were ascertained using the trigger tool.8 In the intervention group as a whole, the raw rate of ADEs decreased from 25.8 per 100 patients per year to 18.3, while in the control group it began at about the same rate (24.3) and stayed about the same (24.8).

The following patient characteristics (extracted from chart review) were considered potential confounders: sex, age, diabetes status, cardiovascular disease status, number of medications, and severity of baseline ADEs. Each was examined for baseline differences between the intervention and control groups. The control group had more diabetes patients (38.1% vs 26.2%, P = .001) and more female patients (67.5% vs 61.0%, P = .052) than the intervention group. Among other examined variables, the differences were not noteworthy.

Table 2

To examine the impact of the intervention, while adjusting for sex and diabetes, ANCOVA was performed, examining the effects of time (pre vs post) and group (intervention vs control), as well as the interaction between time and group. summarizes the ANCOVA results. As can be seen, sex had a signifi cant effect: a higher proportion of female patients in a practice was associated with a higher rate of ADEs. Being in the intervention group versus a control group was associated with a lower rate of ADEs for the practice. The interaction between time (preintervention vs postintervention) and group (intervention vs control) showed a trend toward a decrease in the intervention group compared with the control group over time, but this did not reach the level of statistical signifi cance (P = .104).

Compliance With HEDIS-Recommended Laboratory Monitoring

Rates of compliance with HEDIS-recommended laboratory monitoring were examined for patients on certain chronic medications (ie, angiotensin-converting enzyme inhibitors, diuretics, statins). Each of these medications was used by a high proportion of patients (40%-50%). In both the intervention and control groups, the vast majority of patients (85%-90%) had appropriate lab monitoring for these medications. Chi-square tests showed no signifi cant change in these rates from the preintervention period to the postintervention period, in either group.

Use of the Team Resource Management Tool

Table 3

Response rates to SEMI-P and SAQ-A varied between practices from 58% to 93% of the total practice staff. At each practice, participation in both SEMI-P and SAQ-A included representation from all main staff types, including physicians, physicians extenders (where applicable), nurses, medical assistants, and administrative staff (based on anonymous self-report of job category). At each practice, all available staff participated for at least part of each meeting and all main staff types were represented at each meeting. Each practice prioritized different areas of concern based on their own discussion of SEMI-P and SAQ-A results, and made decisions based on available resources and feasibility. All 4 intervention practices had some successful interventions, as well as others with limited success. All used the system to record and track identifi ed priorities and planned interventions. provides some examples of prioritized medication safety issues, solutions attempted, and the extent of those solutions’ success.

DISCUSSION

The Web-based TRM intervention system was successfully implemented in all 4 intervention sites. Staff in each practice participated in designing and implementing interventions to improve medication safety, tailoring their interventions to their own unique circumstances.

The main outcome was a trend toward a decrease in the rate of ADEs in the intervention group over time, in contrast to the control group, which showed no such trend. This study suggests that the Web-based TRM has the potential to improve medication safety.

Positive safety outcomes can result (particularly in a fragmented healthcare system) only through well-engineered and well-maintained site-specifi c teams. As for any goal-oriented team, motivation and operational coaching are essential, specifically in principles of high reliability organizations and safety enhancement strategies. Support and inculcation of self empowerment are vital for any setting.

This self-empowerment TRM study was a pilot study, and as such, is limited by the small number of practices and the small number of patients included in the outcome measures. The outcome measure used for ADEs was based on chart review and therefore is limited. The sensitivity of the trigger tool method for detecting ADEs is not known but is certainly less than 100%. Therefore, ADE rates determined using this tool should not be seen as complete but only as a subset of the total number of ADEs. For comparability over time, the ascertainment methodology was made consistent from the preintervention to the postintervention period. However, it should be noted that changes in physician charting behaviors may have occurred over time (even though electronic medical records did not change), which may have affected measured ADE rates.

An additional point worthy of discussion is that the risk evaluation that drove the safety improvement process was based on self-refl ection by the practice staff. That is to say, risks were prioritized based on staff perceptions as elicited through their responses to the SEMI-P instrument. This was a deliberate strategy to tap the organizational memory and inculcate ownership of the problems, in keeping with the principles of TRM. An alternative strategy that is more typically used is to employ external experts to perform an independent risk assessment. While these external agents typically lack detailed knowledge of the practice, and therefore may have diffi culty tailoring feasible solutions, they may have the advantage of being trained in quality and safety improvement, which may allow them to make valuable contributions. An interesting avenue that warrants further exploration is the role of practice enhancement associates. These are external agents who internalize themselves into multiple practices over time, becoming a part of each practice’s team. Thus, they may be well suited to provide external evaluation and expertise while also functioning as part of the internal team.

This study achieved its main aims of developing and implementing a Web-based TRM in a variety of ambulatory settings. Future studies should test the intervention on a larger scale and over a longer period of time, with and without practice enhancement associates, and should further explore barriers to change and strategies for overcoming them.

Author Affiliations: From Department of Family Medicine (RS, DA, EM-P, AW, NS, GS), State University of New York at Buffalo; UB School of Management (RS, GS), State University of New York at Buffalo; Dendress Corporation (RB); Buffalo, NY.

Funding Source: Agency for Health Research and Quality (R18 HS 01702).

Author Disclosures: The authors (RS, DA, EM-P, RB, AW, NS, GS) report no relationship or fi nancial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (GS, RS); acquisition of data (DA, EM-P, AW); analysis and interpretation of data (RS, GS, EM-P, DA, AW, NS); drafting of the manuscript (RS, GS); critical revision of the manuscript for important intellectual content (RS, DA, EM-P, RB, AW, NS, GS); statistical analysis (NS, RS); obtaining funding (GS, RS); administrative, technical, or logistic support (RB); and supervision (GS).

Address correspondence to: Gurdev Singh, PhD, 77 Goodell St, Buffalo, NY 14203. E-mail: gsingh4@buffalo.edu.1 . Aspden P, Wolcott JA, Bootman JL, Cronenwett LR, eds; Committee on Identifying and Preventing Medication Errors, Board on Health Care Services, Institute of Medicine. Preventing Medication Errors. Washing ton, DC: National Academies Press; 2007.

2 . Gurwitz JH, Field TS, Harrold LR, et al. Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA. 2003;289(9):1107-1116.

3 . Kohn LT, Corrigan J, Donaldson MS, eds; Committee on Quality of Health Care in America, Institute of Medicine. To Err Is Human:Building a Safer Health System. Washington, DC: National Academies Press; 2000.

4 . Bates DW, Cohen M, Leape LL, Overhage JM, Shabot MM, Sheridan T. Reducing the frequency of errors in medicine using information technology. J Am Med Inform Assoc. 2001;8(4):299-308.

5 . Singh R, Singh A, Servoss TJ, Singh G. Prioritizing threats to patient safety in rural primary care. J Rural Health. 2007;23(2):173-178.

6 . Singh R, Singh A, Taylor JS, Rosenthal TC, Singh S, Singh G. Building learning practices with self-empowered teams for improving patient safety. Journal of Health Management. 2006;8(1): 91-118.

7 . Singh R, Naughton B, Anderson D, McCourt D, Singh G. Building self-empowered teams for improving safety in post operative pain management. In: Henriksen K, Battles JB, Keyes MA, Grady ML, eds. Advances in Patient Safety: New Directions and Alternative Approa ches. Vol 3. Rockville, MD: Agency for Healthcare Research and Quality; 2008:37-50.

8. Singh R, McLean-Plunckett EA, Kee R, et al. Experience with a trigger tool for identifying adverse drug events among older adults in ambulatory primary care. Qual Saf Health Care. 2009;18(3):199-204.

9 . Modak I, Sexton JB, Lux TR, Helmreich RL, Thomas EJ. Measuring safety culture in the ambulatory setting: the safety attitudes questionnaire— ambulatory version. J Gen Intern Med. 2007;22(1):1-5.

10. Singh G, Singh R, Thomas EJ, et al. Measuring safety climate in primary care offi ces. In: Henriksen K, Battles JB, Keyes MA, Grady ML, eds. Advances in Patient Safety: New Directions and Alternative Approaches. Vol 2. Rockville, MD: Agency for Healthcare Research and Quality; 2008:59-72.

11. Colla JB, Bracken AC, Kinney LM, Weeks WB. Measuring patient safety climate: a review of surveys. Qual Saf Health Care. 2005;14(5): 364-366.

Download PDF: IT-Enabled Systems Engineering Approach to Monitoring and Reducing ADEs