Using a Claims Data-Based Sentinel System to Improve Compliance With Clinical Guidelines: Results of a Randomized Prospective Study

Objective: To demonstrate the potential effect of deploying asentinel system that scans administrative claims information andclinical data to detect and mitigate errors in care and deviationsfrom best medical practices.

Methods: Members (n = 39 462; age range, 12-64 years) of amidwestern managed care plan were randomly assigned to anintervention or a control group. The sentinel system was programmedwith more than 1000 decision rules that were capable ofgenerating clinical recommendations. Clinical recommendationstriggered for subjects in the intervention group were relayed totreating physicians, and those for the control group were deferredto study end.

P

P

Results: Nine hundred eight clinical recommendations wereissued to the intervention group. Among those in both groups whotriggered recommendations, there were 19% fewer hospitaladmissions in the intervention group compared with the controlgroup (< .001). Charges among those whose recommendationswere communicated were $77.91 per member per month (pmpm)lower and paid claims were $68.08 pmpm lower than among controlscompared with the baseline values (= .003 for both). Paidclaims for the entire intervention group (with or without recommendations)were $8.07 pmpm lower than those for the entire controlgroup. In contrast, the intervention cost $1.00 pmpm,suggesting an 8-fold return on investment.

Conclusion: Ongoing use of a sentinel system to prompt clinicallyactionable, patient-specific alerts generated from administrativelyderived clinical data was associated with a reduction inhospitalization, medical costs, and morbidity.

(Am J Manag Care. 2005;11:93-102)

Governmental and private initiatives in theUnited Kingdom and the United States^1,2 havecalled for expanded use of information technologyin healthcare and for the incorporation of clinicaldecision support systems³ to detect and alert physiciansto potential medical errors and deviations from bestmedical practices. Previous authors have demonstratedthe value of decision support systems in guiding thechoice of antibiotic therapy⁴ and in monitoring the useof potentially nephrotoxic drugs.⁵ A review of 65 suchsystems demonstrated improved physician compliancewith treatment guidelines in two thirds of studies.⁶However, all previously reported decision support systemshave been deployed within a hospital setting orwithin an integrated delivery system in which electronichealth record systems provide the backbone of clinicalinformation.

sentinel

system

We developed a clinical decision support system thatuses the clinical information contained in administrativeclaims data from physicians, hospitals, pharmacies,and clinical laboratories to identify common errors incare and departures from widely accepted clinicalguidelines. Therefore, the system can operate in anyfee-for-service clinical environment or other environmentin which encounter data are reported withoutrequiring any cooperation from the caregiver communitybeyond responding to clinical recommendations asthey are issued. Because of its ongoing vigilance in monitoringpatient information and spontaneously contactingtreating physicians, we applied the term to distinguish its function from point-of-caredecision support.

The claims-based system we developed is deployedin the care of approximately 4 million Americans whoare enrolled in health plans funded by large employers,managed care organizations, federal employee programs,Medicare managed care programs, and Medicaidprograms. In this report, we describe the results of a 12-month randomized prospective study to test thehypothesis that a claims-driven decision support systemcould increase compliance with evidence-based practices and effect improvements in patient outcomes asmeasured by decreased hospitalization and attendantcost. As subgroup analyses, we examined the causes ofhospitalization for any recommendation issued withsufficient frequency to yield adequate study power.

METHODS

Sentinel System Intervention

International

Classification of Diseases, Ninth Revision

Current Procedural Terminology

The sentinel system was designed as a rule-basedartificial intelligence engine combined with an automaticmessage generator that conveys clinical recommendationsand supporting literature to treating physicians.Software coding was programmed in a combination ofC++ and Visual Basic (Microsoft Corporation, Redmond,Wash) that draws data from an Oracle 8i-based datawarehouse (Oracle Corporation, Redwood City, Calif).Daily data inputs include physician-generated insuranceclaims, hospital discharge and outpatient claims,laboratory claims and laboratory test results, and pharmacyclaims. Information is coded in , ClinicalModification codes, codes, Logical Observation Identifier Names and Codes,and National Drug Code identifiers.

Candidate clinical issues for inclusion in the clinicalrules engine were identified from national multicenterclinical trials, guidelines published by the federal governmentor medical specialty societies, and Food andDrug Administration-approved pharmaceutical labelingby an in-house committee of physicians, working inconsultation with health plan medical directors and apanel of medical consultants. Typical issues targeted bythe rules engine include the following: (1) absence ofangiotensin-converting enzyme inhibitor therapy inpatients with congestive heart failure^7-9 and in those whomeet the Heart Outcomes Prevention Evaluation(HOPE) trial criteria,¹⁰ (2) absence of &#946;-blocker use inpatients who have had myocardial infarction,^11,12 (3)absence of anticoagulation in patients with atrial fibrillationand structural heart disease,^13,14 and (4) absenceof documented laboratory monitoring in patients takingwarfarin sodium, glitazones,¹⁵ and other medicationsthat require specific laboratory tests. Depending on theclinical urgency of the issue, notification may be byphysician-to-physician telephone contact, fax, or letter.

The criteria that trigger the engine to issue a specificrecommendation are multidimensional decisionmatrices that take into account all available informationon each patient. For instance, the criteria surroundingthe use of &#946;-blockers after myocardial infarctionattempt to identify known contraindications for &#946;-blocker therapy, such as severe bronchospasm andadvanced heart block in the absence of a pacemaker.Should any of those contraindications be identified, therecommendation is not issued.

The system contains more than 1000 decision matricesthat, when triggered, result in the transmission of acommunication to the treating physician, recommendingthat he or she consider a particular avenue of care.Because a claims-based system such as this will neverhave as much information about the patient as thetreating physician, all recommendations make clearthat the communication is merely for the physician'sconsideration and that there may be mitigating circumstancesthat might render the recommendation inappropriate.For this reason, such communications areinterchangeably referred to as care considerations.

Patients and Setting

The study was performed among the commerciallyinsured population of a university-affiliated managedcare plan, in which patients were free to choose theirprimary care physician and to move their care from oneprimary care physician to another. The protocol wasapproved by the human subjects committee of thehealth plan before inception and was in accord with theDeclaration of Helsinki with regard to protection ofhuman subjects. In the annual enrollment form, all planmembers provided written informed consent to theanalysis of their healthcare data and to the medicalmanagement and quality improvement activities of thehealth plan. Eligible subjects were notified by letter thatthe health plan was undertaking a quality improvementstudy and that they might be randomly allocated to aprogram in which their physicians might receive informationrelevant to their care and were given a telephonenumber to call if they wished not to be includedin such a study. To maximize the study power, interventionand control group members consisted of allhealth plan enrollees who were between the ages of 12and 64 years and had incurred at least 1 physician claimor 1 pharmacy claim in the 12 months before enrollment.Patients were assigned to an intervention or acontrol group, using an individually assigned randomnumber. Assignment occurred on a single date at studyinception. Neither patients nor treating physicians wereinformed of the allocation, although it is likely thatthose physicians who received communications aboutspecific patients surmised that those patients were partof the intervention group.

Study Power and Sample Size

The study was designed to measure potential differencesin cost of care and to measure the extent to whichphysicians comply with clinical prompts issued in thismanner. The sample size required was calculated to havea power of greater than 80% (&#946;= .2) to detect a 5% differencein claims cost and compliance with recommendationsbetween the 2 groups at a type I error rate of .01(2-sided). We determined that subgroup analysis (withthe same study power) was feasible for any recommendationissued to at least 100 intervention and controlgroup subjects in which the adverse event rate in controlsexceeded 40%.

Data Analysis

All analyses were performed on an intentionto-treat basis. For each group, individual members'claims were measured monthly during thestudy and expressed in terms of dollars permember per month (pmpm). All claims were alsobroken down into type of service, including inpatient,outpatient, professional services, and pharmacy.For each group, the totals and means werecalculated for total claims and claims by type ofservice. We also compared the costs for the interventiongroup participants who triggered careconsiderations vs controls who triggered butwere not issued care considerations.

z

t

P

P

To compare differences in total admissions andtotal days in the hospital, 2×2 tables were createdcontaining the numbers admitted and the numbersnot admitted in the intervention and controlgroups (as well as for days in the hospital and daysnot in the hospital) and were tested by &#967;² test.Differences in the rates of hospital admissions per1000 subjects were tested by test. Differences inmean charges and paid amounts in the interventionand control groups were tested by test,weighted for the number of months under observation.< .02 was required for rejection of thenull hypothesis. As a safety measure, a maskedanalysis was conducted at 6months to assess whether therewas any difference (< .05) inthe number of deaths betweenthe 2 groups.

RESULTS

A total of 39 462 subjectswere initially enrolled in thestudy (Figure 1). At baseline,no significant differences wereobserved in age or sex betweenthe intervention and controlgroup subjects (Table 1).Eighty-four percent of control group members and 85%of intervention group members remained in the studyfor the entire 12 months (Figure 1). Approximately halfof the early withdrawals occurred in the first month,comprising individuals enrolled and randomized inDecember who chose another health plan for the studyyear. Therefore, 90% of control and intervention groupmembers who began the study remained under observationfor the entire study. During the study year, no differencein mortality was observed between theintervention group and the control group. The data availableto us identified hospital mortality, including deathon arrival to the hospital, and mortality documented bythe health plan.

Compliance With Recommendations

P

Nine hundred eight clinical recommendations wereissued to the intervention group: 395 of those involvedthe recommendation to start a medication, 150entailed the suggestion to stop a medication, and 790suggested a test or procedure (Table 2). Compliancewith care considerations that entailed starting a newdrug was measured by ascertaining the date the drugwas dispensed at the pharmacy. Physicians compliedwith 24% of these "add-a-drug"recommendations inthe intervention group. In the control group, physiciansspontaneously instituted the treatment thatwould have been recommended in 17% of instances inwhich the recommendation was triggered but notissued. This 42% relative difference in compliance wasstatistically significant (= .007). The same clinicaltriggers that generated add-a-drug recommendationswere measured at baseline, and identical rates ofspontaneous resolution of those clinical situations inthe absence of intervention were noted in those subjectsultimately assigned to the intervention groupand to the control group (18% for both).

P

As seen in Table 2, more recommendations weretriggered for the intervention group than for the controlgroup. Although this difference was statisticallysignificant (< .01), it was associated with recommendationsto obtain laboratory tests. This recommendationcategory is unrelated to differences in costor hospital utilization (Figure 2). We believe that themost likely explanation for this is that the care considerationengine was run weekly on the interventiongroup patients, whereas it was run at the end of thestudy on the control group patients. This was to avoidthe potential ethical problem of withholding actionableclinical information from control group participants.

Compliance with recommendations could notaccurately be assessed in the case of recommendationsto discontinue a medication, because discontinuancedoes not result in a medical order. Compliancewith recommendations for laboratory testing was notascertainable, in part because approximately onethird of laboratory testing, including all inpatient laboratorytesting, in this care system did not generate adiscrete claim.

Observed Differences inHospital Utilization

P

P

Overall, there were 115 more hospital admissionsin the control group than in the intervention group(Table 3). Most of this difference (96 admissions)occurred in the 5% of subjects who triggered recommendations.No meaningful differences were observedin the 95% of the population that did not trigger recommendations.Among all those triggering recommendationsin the intervention and control groups,19% fewer hospital admissions per 1000 subjectswere observed in the intervention group (< .001).Because intervention group members who were admittedexperienced a slightly longer stay, the overall differencein hospital bed days per 1000 subjects wasonly 8% (= .004).

Observed Differences in Chargesand Paid Claims

For total charges and paid claims, differencesbetween the intervention and control groups were notsignificant at baseline but became highly significantduring the study (Figure 3). Although charges in thestudy year compared to the baseline were higher inboth groups, charges in the intervention group were$18.62 pmpm (95% confidence limits, $12.00-$25.00pmpm) lower and paid claims were $8.07 pmpm (95%confidence limits, $5.00-$11.00 pmpm) lower. The differencein charges was observed for all categories ofcare except pharmacy claims, in which a small differencewas observed in favor of the control group (pharmacyclaims increased slightly for both groups frombaseline to the study year because of changes in reportingacross the health plan). This increase in observedpharmacy related charges is attributable to implementationof the add-a-drug recommendations. Multivariateanalyses that included baseline utilization as an independentvariable did not attenuate this effect.

Subgroup Analyses of Evidence forMechanism of Action

Examination of Those Who Triggered Care Considerations.

P

Examination of the subgroup that triggeredcare considerations reveals a more striking differencebetween the intervention group and controls. Figure 4demonstrates that a far greater financial differencebetween the intervention group and controls is observablein the subpopulation of those who triggered careconsiderations. Comparing the study year results withbaseline values, the intervention group patientsincurred $77.91 pmpm (95% confidence limits, $26.00-$130.00 pmpm) less in charges and $68.08 pmpm (95%confidence limits, $39.00-$98.00 pmpm) less in paidclaims than controls (= .003 for both). There was nostatistically significant difference between interventionand control group subjects who did not trigger careconsiderations.

Examination by Type of Care Consideration

Issued.

When differences in utilization are analyzed bytype of care consideration issued (Figure 2), the mostpronounced difference is associated with those patientswho triggered recommendations to add a specific medication.A smaller but meaningful difference was identifiablein patients who triggered recommendations tostop specific medications. No significant difference wasidentified in cost or utilization among patients who triggeredrecommendations for laboratory or diagnosticprocedures.

Analysis of Heart Outcomes Prevention Evaluation

Trial-Related Hospitalizations.

The only clinical recommendationthat was issued with sufficient frequencyto enable subgroup analysis of outcomes was the recommendationto prescribe angiotensin-converting enzymeinhibitors in patients meeting the HOPE trial criteria.¹⁰ Inthese 311 patients (156 intervention and 155 controlgroup subjects), a $510.00 pmpm difference in totalcharges was observed between the control and interventiongroups (date not shown), as well as a difference of 12hospital admissions per 100 patients (Table 4).

The frequency with which this HOPE recommendationwas issued afforded the ability to examine the clinicaldetails associated with the hospital admissions todetermine whether the difference in hospitalization wasrelated to the intervention. We considered hospitaladmissions for stroke, cardiac disease (except for dysrhythmia),and vascular complications of diabetes mellitusto be consistent with the clinical outcomes studiedin the HOPE trial. Hospital admissions for other diseaseentities were not considered to be plausibly preventedby the addition of angiotensin-converting enzymeinhibitors.

P

z

There were 152 admissions during the study yearamong the 155 controls who triggered the recommendationto prescribe angiotensin-converting enzymeinhibitors, compared with 133 admissions among theintervention group members who triggered this recommendation(Table 4). Of those, 69 control group admissions,compared with 49 intervention group admissions,were for cerebrovascular or cardiovascular disease. Thisdifference of 20 admissions (45% vs 37%, = .02) was statisticallysignificant by test. In contrast, there was nodifference in the rate of hospital admission for diagnosesunrelated to the HOPE trial end points. This reductionin the risk of hospitalization was consistentwith the reduction in the riskobserved in the HOPE trial.¹⁰

Return on Investment

The study was not intended as aformal cost-effectiveness or cost savingsanalysis in that we did not directlymeasure costs at the patient orcaregiver level, nor did we considernoneconomic costs or benefits. Ourstudy can, however, speak to thereturn on investment associated withthe intervention from the perspectiveof the payer. The interventioncost ranged from $1.00 to $1.50pmpm, depending on the demographicsof the covered population,with older populations generatinghigher charges because of greaterseverity of illness. In the case of theintervention population, the interventioncost was $1.00 pmpm andthe overall return was $8.07 pmpm,indicating an 8-fold return oninvestment in the first year of theintervention. For many of the interventionstriggered, one would expectongoing benefits to accrue, suggestingthat over time the return oninvestment may be even greater.

DISCUSSION

Our findings suggest that deploymentof a sentinel system that issuesclinical prompts based on administrativelyderived clinical data has thepotential to positively influence physicianpractice and may be associatedwith decreases in morbidity andhealthcare cost among patients withtargeted conditions. The use of physicianprompts, delivered primarily bywritten communication, was associatedwith a moderate degree (42%) ofincreased compliance with widelyaccepted standards of care, compared with thatobserved in a randomly allocated control group. Theintervention cost $1.00 pmpm to deploy and was associatedwith lower charges of $18.62 pmpm and lowerpaid claims of $8.07 pmpm in the intervention groupcompared with controls.

The difference in compliance with standards of careassociated with our intervention was clinically and statisticallysignificant and was associated with a substantialdifference in hospitalization and cost of care.However, the compliance, even in the interventiongroup (24%), leaves significant room for improvement.More potent interventions are needed that may involvefurther engaging the physician via electronic means andpoint-of-care systems, in addition to engaging thepatient directly.

Two factors argue in favor of a causal relationshipbetween the intervention and the observed outcome and,hence, its clinical plausibility. First, the observed differenceis almost completely attributable to those membersof the intervention and control groups who triggered recommendations.Second, the subgroup analysis of patientswho triggered the recommendation to prescribeangiotensin-converting enzyme inhibitors for subjectsmeeting the HOPE criteria demonstrated that the entiredifference in hospitalization was attributable to admissionfor cardiac and peripheral vascular disease.

It may seem counterintuitive that intervening 5% ofthe population can have such a broad effect. However,one must remember that most commercially insuredpersons have no major illnesses. Most hospital utilizationis attributable to the few with chronic disease. Bydrawing on the existing stream of administrative data,we constructed an approach that does not depend onunderlying health information infrastructure (otherthan ubiquitous claims submission systems) and thatdraws data from a broad array of otherwise unconnectedcaregivers.

Although this study addressed the potential benefitof deploying a sentinel system in a commercial population,the disease entities most affected by the intervention—neurological, respiratory, and cardiovascularconditions—are the major drivers of health cost in olderpopulations. Therefore, we expect that sentinel systemssuch as this wouldhave an even greatereffect when deployedin populations withgreater morbidity,such as populationscovered by Medicaidand Medicare.

Limitations of theIntervention

In an ideal world,clinical decision supportsystems wouldbe embedded withincomprehensive electronichealth informationenvironmentsthat include medicalrecords and orderentry. The best momentto intervene and prompt a caregiver to complywith evidence-based guidelines is at the point of care,using detailed clinical information about the patient.Physicians are likely to be more responsive to promptsgenerated at the point of care, provided that thoseprompts are clinically on target and patient specific.Unfortunately, few Americans (except for active-dutymilitary and veterans) are cared for in an electronichealthcare environment that enables point-of-caresystems, and those systems may not yet have developedthe instantaneous decision analytic capabilitiesnecessary to address the broad range of issues affectingthe ill.³⁴

Today, most Americans are cared for in a communityprovider-based environment. Although point-of-caresystems are available for installation in some tightly integratedhealthcare delivery systems, the exchange ofhealth information among nonrelated medical providersrequires currently unavailable solutions to technical andlegal barriers. Outside of an integrated delivery system,only a claims-based system has the potential to integrateclinical data from the patient's entire network ofproviders, pharmacies, and laboratories. We, like muchof the readership, dream of a future in which electronicmedical records flow seamlessly from one point of careto another. In that future, we believe that sentinel systemswill be integral to comprehensive point-of-care systemsand will draw on much richer sources of data.

The intervention that we implemented lags days toweeks behind the moment of care and is restricted tosummary information contained in diagnosis and procedurecodes, along with pharmacy data and laboratory values. This lag may reduce compliance with careconsiderations and limits the type of issues that maybe addressed to clinical issues involving the prescriptionof medications for chronic illness, monitoring ofthose medications, and management of the underlyingillness. Medication issues that are likely to have animmediately harmful effect, for instance, cannot usefullybe addressed in a system of this type. However, asis evident from our data, there are many such issuesthat can be addressed, even in a healthy working-agepopulation.

Our system relies on the diagnosis and procedurecodes contained in administrative claims data, theaccuracy of which is far from perfect. The logic algorithmswithin the sentinel system attempt to maximizethe specificity of the inferences gained from claims databy requiring multiple pieces of corroborating information(eg, diagnosis codes indicative of diabetes mellitus,evidence of pharmacotherapy for diabetes mellitus, andlaboratory tests indicative of hyperglycemia) beforeclassifying a patient as having a particular condition.Nevertheless, as the specificity of the data streamimproves, particularly through the widespread implementationof electronic health records, sentinel systemswill become more clinically useful.

Limitations of the Study

Our study is subject to at least 3 sources of bias,including selection bias, treatment bias, and ascertainmentbias. We designed this study as a community-basedtrial, in which the study cohort was asrepresentative as possible of the underlying community.The only potential selection bias is the extent towhich the membership of the health plan studied is oris not representative of the commercially insured populationof the Cleveland metropolitan area. Given thatenrollment is employer based, there is little reason tobelieve that this represents a significant bias.

A second potential source of bias is treatment bias.Once a physician received a clinical recommendation,the physician was then alerted to the fact that thepatient in question was under scrutiny. If this promptedphysicians to rethink other aspects of the patient'scare, there may have been an effect larger than the simpleeffect of implementing that specific recommendation.Because the objective of this study was todemonstrate the potential value of sentinel systems inimproving overall care, not the effect of any specificclinical recommendation, this bias (if it was a factor)does not diminish the overall value of the findings ofthe study. One alternative design would have been toattempt a nonspecific notification of control groupphysicians to try to attenuate this bias.

The study is also subject to ascertainment bias inthat it is based on surrogate outcome measures, namely,hospitalization and other provision of medical care,rather than ultimate outcomes, such as quality of life orsurvival. That limitation is inherent in nearly everystudy that attempts to measure health system effectsrather than the traditional end points associated withclinical trials.

CONCLUSIONS

Our study demonstrates that decision support systemscan be applied across a system of care to improvecompliance with evidence-based medicine. The resultssuggest that such compliance improves outcomes ofcare as suggested by the clinical trials that underlie theindividual clinical recommendations. The study wasperformed using the summary data available in anadministrative claims stream. As richer electronichealth records become available across the medicalenterprise, sentinel systems such as this will haveincreased clinical accuracy and deliver patient-specificrecommendations in an increasingly timely manner.

Acknowledgment

The authors want to thank James B. Rebitzer, PhD, of Case WesternUniversity for methodological and statistical guidance and expertise.

From Active Health Management, Inc, New York, NY (JCJ, GS, JJ, IJ, LR); PotomacInstitute for Policy Studies, Arlington, Va (JCJ); Qualchoice, Inc, and Case Western ReserveUniversity School of Medicine, Cleveland, Ohio (TL); and American Re-InsuranceCompany, Princeton, NJ (JBC).

This study was funded by American Re-Insurance Company, which provides servicesto Qualchoice, Inc. The role of American Re-Insurance Company was limited to review ofthe study protocol before study inception. The company had no contact with those implementingthe clinical interventions or with the physicians who received the interventionrecommendations.

Dr Javitt is a shareholder of Active Health Management, Inc, and remains a consultantto the company. Drs Steinberg, Juster, and Reisman are employees and shareholders ofActive Health Management, Inc. Although not at the time of the study, Dr Couch independentlycontracts with Active Health Management, Inc. He is now with New WorldHealthcare Solutions, Princeton, NJ.

Address correspondence to: Jonathan C. Javitt, MD, MPH, 1700 Pennsylvania Avenue,Suite 400, Washington, DC 20006. E-mail: jjavitt@healthdirections.net.

Crossing the Quality Chasm: A New Health System for

the 21st Century.

1. Institute of Medicine. Washington, DC: National Academy Press; 2001.

Report to the

President: Transforming Health Care Through Information Technology.

2. President's Information Technology Advisory Committee. Arlington,Va: National Coordination Office for Information Technology Research andDevelopment; 2001.

Key Capabilities of an Electronic Health Record.

3. Institute of Medicine. Washington, DC: National Academy Press; July 31, 2003.

N Engl J Med.

4. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted managementprogram for antibiotics and other antiinfective agents. 1998;338:232-238.

Arch Intern Med.

5. Rind DM, Safran C, Phillips RS, et al. Effect of computer-based alerts on thetreatment and outcomes of hospitalized patients. 1994;154:1511-1517.

JAMA.

6. Hunt DL, Hanes RB, Hanna SE, Smith K. Effects of computer-based clinicaldecision support systems on physician performance and patient outcomes.1998;280:1339-1346.

N Engl J Med.

7. SOLVD Investigators. Effect of enalapril on survival in patients with reducedleft ventricular ejection fraction and congestive heart failure. 1991;325:293-302.

N Engl J Med.

8. SOLVD Investigators. Effect of enalapril on mortality and the development ofheart failure, I: asymptomatic patients with reduced left ventricular ejection fractions.1992;327:685-691.

N Engl J Med.

9. CONSENSUS Trial Study Group. Effects of enalapril on mortality in severe congestiveheart failure: results of the Cooperative North Scandinavian EnalaprilSurvival Study. 1987;316:1429-1435.

N Engl J Med.

N Engl

J Med.

N Engl J Med.

10. Yusuf S, Sleight P, Pogue J, Bosch J, Davies R, Dagenais G; Heart OutcomesPrevention Evaluation Study Investigators. Effects of an angiotensinconverting-enzyme inhibitor, ramipril, on cardiovascular events in high-riskpatients [published corrections appear in 2000;342:748 and 2000;342:1376]. 2000;342:145-153.

JAMA.

11. beta-Blocker Heart Attack Study Group. A randomized trial of propranolol inpatients with acute myocardial infarction, I: mortality results. 1982;247:1707-1714.

JAMA.

12. beta-Blocker Heart Attack Study Group. A randomized trial of propranolol inpatients with acute myocardial infarction, II: morbidity results. 1983;250:2814-2819.

JAMA.

13. Ezekowitz MD, Levine JA. Preventing stroke in patients with atrial fibrillation.1999;281:1830-1835.

Arch Intern Med.

14. Atrial Fibrillation Investigators. Risk factors for stroke and efficacy of antithrombotictherapy in atrial fibrillation: analysis of pooled data from five randomizedcontrolled trials. 1994;154:1449-1457.

15. Physicians' Desk Reference. 57th ed. Montvale, NJ: Thompson PublishingCompany; 2003:645, 995, 1473, 2510, 3180.

J Am Coll Cardiol.

16. Pasternak RC, Smith SC Jr, Bairey-Merz CN, Grundy SM, Cleeman JI, LenfantC; American College of Cardiology; American Heart Association; National Heart,Lung, and Blood Institute. ACC/AHA/NHLBI clinical advisory on the use and safetyof statins. 2002;40:567-572.

Diabetes Care.

17. American Diabetes Association. Position statement: standards of medical carefor patients with diabetes mellitus. 2002;25(suppl 1):S33-S49.

N Engl J Med.

18. Bailey CJ, Turner RC. Metformin. 1996;334:574-579.

Chest.

19. American College of Chest Physicians. Proceedings of the American College ofChest Physicians 5th Consensus on Antithrombotic Therapy: 1998. 1998;114(suppl):439S-769S.

N Engl J Med.

N Engl J Med.

20. Radermacher J, Chavan A, Bleck J, et al. Use of Doppler ultrasonography topredict the outcome of therapy for renal-artery stenosis. 2001;344:410-417.21. Safian RD, Textor SC. Renal-artery stenosis. 2001;344:431-442.

Guidelines for the Diagnosis and Management

of Asthma: Expert Panel Report 2.

22. National Institutes of Health. Bethesda, Md: National Heart, Lung, and BloodInstitute, Public Health Service, US Dept of Health and Human Services; 1997.

23. National Institutes of Health. Osteoporosis prevention, diagnosis, and therapy.NIH Consens Statement. 2000;17:1-45.

Arthritis Rheum.

24. American College of Rheumatology Ad Hoc Committee on Glucocorticoid-Induced Osteoporosis. Recommendations for the prevention and treatment of glucocorticoid-induced osteoporosis: 2001 update. 2001;44:1496-1503.

Harrison's Textbook of Internal Medicine.

25. Wood AJ. Adverse reactions to drugs. In: Braunwald E, Fauci AS, Kasper DL, etal, eds. 15th ed. Philadelphia, Pa: CBSSaunders Publishers Inc; 2001:436.

N Engl J Med.

26. Brown BG, Zhao XQ, Chait A, et al. Simvastatin and niacin, antioxidant vitamins,or the combination for the prevention of coronary disease. 2001;345:1583-1592.

JAMA.

27. Expert Panel on Detection, Evaluation, and Treatment of High BloodCholesterol in Adults. Executive summary of the Third Report of the NationalCholesterol Education Program (NCEP) Expert Panel on Detection, Evaluation, andTreatment of High Blood Cholesterol in Adults (Adult Treatment Panel III). 2001;285:2486-2497.

Cleve Clin J Med.

28. Younossi ZM. Viral hepatitis guide for practicing physicians: Cleveland ClinicMedicine. 2000;67(suppl 1):S16-S45.

N Engl J

Med.

29. Gaspoz JM, Coxson PG, Goldman PA, et al. Cost effectiveness of aspirin,clopidogrel, or both for secondary prevention of coronary heart disease. 2002;346:1800-1806.

Hematol Oncol Clin

North Am.

30. Whiteman T, Hassouna HI. Hypercoagulable states. 2000;2:355-377, viii.

J Am Coll Cardiol.

31. Hunt SA, Baker DW, Chin MH, et al; American College of Cardiology/AmericanHeart Association. ACC/AHA guidelines for the evaluation and managementof chronic heart failure in the adult: executive summary: a report of the AmericanCollege of Cardiology/American Heart Association Task Force on Practice Guidelines(committee to revise the 1995 Guidelines for the Evaluation and Managementof Heart Failure). 2001;38:2101-2113.

Circulation.

32. Fuster V, Ryden LE, Asinger RW, et al; American College ofCardiology/American Heart Association Task Force on Practice Guidelines;European Society of Cardiology Committee for Practice Guidelines and PolicyConferences (Committee to Develop Guidelines for the Management of PatientsWith Atrial Fibrillation); North American Society of Pacing andElectrophysiology. ACC/AHA/ESC guidelines for the management of patients withatrial fibrillation: executive summary: a report of the American College of Cardiology/American Heart Association Task Force on Practice Guidelines and theEuropean Society of Cardiology Committee for Practice Guidelines and PolicyConferences (committee to develop guidelines for the management of patients withatrial fibrillation) developed in collaboration with the North American Society ofPacing and Electrophysiology. 2001;104:2118-2150.

Cochrane

Database Syst Rev.

33. Koudstaal PJ. Anticoagulants for preventing stroke in patients with nonrheumaticatrial fibrillation and a history of stroke or transient ischemic attacks. 2000:CD000185.

Revolutionizing

Health Care Through Information Technology.

34. President's Information Technology Advisory Committee. Arlington, Va: NationalCoordination Office; 2004.

Download PDF: Using a Claims Data-Based Sentinel System to Improve Compliance With Clinical Guidelines: Results of a Randomized Prospective Study