Evaluation of Electronic Medical Record Administrative Data Linked Database (EMRALD)

Background: Primary care electronic medical records (EMRs) represent a potentially rich source of information for research and evaluation.

Objective: To assess the completeness of primary care EMR data compared with administrative data.

Study Design: Retrospective comparison of provincial health-related administrative databases and patient records for more than 50,000 patients of 54 physicians in 15 geographically distinct clinics in Ontario, Canada, contained in the Electronic Medical Record Administrative data Linked Database (EMRALD).

Methods: Physician billings, laboratory tests, medications, specialist consultation letters, and hospital discharges captured in EMRALD were compared with health-related administrative data in a universal access healthcare system.

Results: The mean (standard deviation [SD]) percentage of clinic primary care outpatient visits captured in EMRALD compared with administrative data was 94.4% (4.88%). Consultation letters from specialists for first consultations and for hospital discharges were captured at a mean (SD) rate of 72.7% (7.98%) and 58.5% (15.24%), respectively, within 30 days of the occurrence. The mean (SD) capture within EMRALD of the most common laboratory tests billed and the most common drugs dispensed was 67.3% (21.46%) and 68.2% (8.32%), respectively, for all clinics.

Conclusions: We found reasonable capture of information within the EMR compared with administrative data, with the advantage in the EMR of having actual laboratory results, prescriptions for patients of all ages, and detailed clinical information. However, the combination of complete EMR records and administrative data is needed to provide a full comprehensive picture of patient health histories and processes, and outcomes of care.

Am J Manag Care. 2014;20(1):e15-e21This study examined the comprehensiveness of primary care electronic medical record (EMR) data compared with administrative data.

• Primary care EMRs have reasonably comprehensive capture of primary care physician visits, laboratory tests, and prescriptions by the EMR physician.

• Laboratory tests and prescriptions by other providers may be missing; there is incomplete communication with specialists, and particularly poor communication between EMRs and hospitals.

• Until integrated systems with open communication channels between all aspectsof the healthcare system are in place, the most comprehensive patient health histories can be achieved by combining information from complete EMR records and administrative data.

Although the United States and Canada have historically lagged behind other industrialized countries in the adoption of electronic medical records (EMRs) in primary care,¹ with the introduction of the Health Information Technology for Economic and Clinical Health Act in 2009 in the United States² and the establishment of Canada Health Infoway³ and provincial EMR adoption support programs in Canada, the uptake of EMRs in both countries is rapidly increasing.⁴ This development, especially in primary care physician practices, has resulted in a new, potentially rich source of clinical information not only for point-of-care clinical practice but also for secondary purposes such as research and quality performance evaluation.

Because of its single-payer healthcare system, Canada has comprehensive health-related administrative databases that cover the entire population. Complete, provincewide, population-level administrative databases have been shown to be highly accurate in capturing hospitalizations and prescriptions for Ontario residents 65 years and older.⁵ Also in these databases, physician billing data accurately capture frequency of patient encounters, but the depth and details of patient clinical encounters are unavailable. Indeed, a previous study in the United States found that EMR data in community health centers were more complete than Medicaid claims data for assessing diabetes preventive care.⁶

Because use of primary care EMR data in Canada for secondary purposes is in its relative infancy, we set out to determine the completeness and comprehensiveness of the EMR data compared with administrative data.

METHODS

We developed the Electronic Medical Record Administrative data Linked Database (EMRALD) at the Institute for Clinical Evaluative Sciences (ICES). The Institute for Clinical Evaluative Sciences is a “prescribed entity” under provincial privacy legislation and thus is able to collect individual-level patient health information without patient consent based on policies and procedures in place to protect patient privacy and confidentiality.⁷ EMRALD was developed using data from family physicians (FPs) in Ontario using Practice Solutions EMR, the market-leading EMR software vendor in Ontario.⁸ All clinically relevant data are extracted from the participating physician’s EMR and each patient is anonymously linked, using their scrambled health card number, to the health-related administrative databases for the province of Ontario housed at ICES.

Physicians participate on a completely voluntary basis and are required to have had their EMR a minimum of 2 years to ensure that the EMR is adequately populated. Over the past decade Ontario has undergone major primary care reform such that the majority of all FPs in the province practice under one of the reform models that require “rostering” of patients (identification of patients with the 1 primary care provider who is most responsible for their care).⁹ All of the EMRALD FPs practice under one of the various primary care reform models of care. This analysis was confined to data that were captured in 2008 for rostered patients of participating FPs who had at least 1 visit to their physician in the 2 years before January 1, 2008.

Family Physician Clinic Visits

To determine the capture of all FP visits by patients rostered to EMRALD physicians (regardless of whether the visit was to an EMRALD physician), FP outpatient visits billed to the Ontario Health Insurance Plan (OHIP) physician billing database were compared with visits recorded in the EMR to determine the capture of FP visits by patients rostered to EMRALD physicians. Visit comparison between OHIP and EMRALD was confined to EMRALD physician OHIP visits only, to assess how completely EMRALD physicians were documenting patient visits within their EMR.

Specialist Consultations and Hospitalizations

To determine the capture of specialist visits and hospitalizations within EMRALD, OHIP billings by specialists performed in an office setting for an initial consultation and hospitalizations, as recorded in the Canadian Institute for Health Information Discharge Abstract Database, were compared with consultation letters and hospital discharge summaries in EMRALD within 14 days and within 30 days of the specialist billing date or the hospital discharge date.

Laboratory Data

A frequency count of OHIP laboratory fee codes identified the 20 most common laboratory tests ordered. Comparisons were made between the laboratory tests recorded on the same day in EMRALD and in OHIP.

Prescription Data

The EMR records when a drug is prescribed to the patient by the EMR physician, and the Ontario Drug Database (ODB) records when a drug was dispensed at a pharmacy (regardless of the prescriber). The EMR may also list drugs prescribed to a patient by other providers (ie, specialists), but only if such data are manually updated by the EMR-using physician; thus, the EMR is variably populated with these data. The prescription field within EMRALD and the prescriptions dispensed in ODB in 2008 were used to compile 2 drugs lists for the top 50 dispensed Canadian medications,10 grouped into 7 clinical categories. The 2 lists for each patient were compared. This analysis was confined to patients 65 years and older as of January 1, 2008, as the ODB only captures drugs universally for all Ontario residents 65 years and older.

Match Rates

Match rates were calculated for all parameters of comparison by calculating the mean match rate and standard deviation (SD) for all the clinics, giving each clinic equal weighting regardless of size.

This study received ethics approval from Sunnybrook Health Sciences Research Ethics Board.

RESULTS

Overall, there were 56,107 patients with an average age of 39.7 years from 54 physicians practicing in 15 geographically distinct clinics. The mean years since graduation of the physicians was 19.3 years (SD = 10.4 years). Seventy percent of the physicians were in urban practice, 56% were male, and 98% were in group practice. The average duration of time on the EMR was 4.5 years (SD = 2.6 years).

Family Physician Visits

Nearly 80% of the FP billings in OHIP that occurred in an office setting were captured in EMRALD. To determine whether the remaining billings were missing because patients were seen elsewhere (eg, a walk-in clinic) and therefore their visits were not recorded in the EMR, we limited the comparison to OHIP billings made by EMRALD physicians. In that case, we found that nearly all the billings for patients had a corresponding progress note entry in the EMR. Thus, we concluded that the approximately 15% of billings missing from the EMR were because patients were seen elsewhere and that the EMRALD participating physicians were fully using the EMR to record their patient encounters (Table 1).

Specialist Clinical Encounters and Hospitalizations

On average, just over two-thirds of initial specialist visits resulted in a consultation letter captured in EMRALD within 14 days of the visit; this percentage increased by less than 5% when the interval was expanded to 30 days (Table 1). Just over half of the hospital discharges had documentation of the hospitalization within EMRALD (Table 1).

Laboratory Tests and Prescriptions

We found that on average approximately three-fourths of the laboratory tests captured in EMRALD were also billed in OHIP, and most clinics had a higher match rate. When the laboratory tests recorded in OHIP were compared with what was recorded in EMRALD, approximately two-thirds of the tests billed to OHIP were captured (Table 1). Urinalysis tests had the poorest capture of all the laboratory tests in OHIP compared with EMRALD (Table 2).

Capture of prescriptions in EMRALD compared with drugs dispensed in ODB was high. However, only just over two-thirds of the drugs dispensed in ODB were captured in EMRALD (Table 1). Antibiotics (as represented by amoxicillin) were the drug class that had the poorest capture rate. However, the capture rates for drugs to treat chronic conditions such as cardiovascular disease and endocrine diseases appeared to be relatively high (Table 3).

DISCUSSION

We found that primary care EMR data capture compared well with administrative data. Like previous US studies comparing EMR data with administrative data,^6,11 our results support the idea that primary care process quality measures involving laboratory test ordering or prescriptions are better assessed through the EMR than through administrative data because the EMR is more likely to reflect what the primary care physician ordered or prescribed.

Although our results may seem to indicate that administrative data have superior capture of laboratory results, both administrative data and the EMR have their weaknesses. Unlike administrative data, the EMR has actual laboratory results available. However, the EMR often misses tests that were not ordered by the primary care provider. There was a larger SD for the lab results compared with the other assessed data elements because 2 clinics had much lower than average match rates. These 2 low-match-rate clinics both had far more laboratory test data in the EMR than in the administrative data and were likely clinics where the patients used a hospital-based laboratory, which is not currently captured in provincial administrative databases.

Prescription data within the EMR contain prescriptions for patients of all ages (rather than just the elderly and/or those on social assistance, as in our provincial drug database), whereas the administrative data provide information on medications dispensed regardless of the prescriber. Because administrative data show the prescriptions dispensed while the EMR data shows the prescriptions prescribed, the match rates should be considered in light of previous studies indicating that approximately 10% of prescriptions do not get filled.¹² The relatively low capture rate for antibiotics may reflect the use of walk-in clinics or the emergency department for the treatment of acute infectious conditions. An additional advantage of EMR data is the capture of prescribed medications that are over the counter or that are not on the provincial drug formulary.

We found that nearly three-fourths of specialist consultation letters were captured in the EMR, but hospital discharge summaries were less than optimally captured. These findings are similar to previous findings in the United States¹³ and highlight the need for improvements in communications between providers to ensure completeness of primary care EMRs.

Uncertainty regarding generalizability of our findings arises from the variability across provider users and across EMR systems.¹⁴ Each EMR software package has its own unique data structure. Until Canadian standards are developed, implemented, and enforced for entering and exporting EMR data elements, the quality of the data captured and the ability to identify discrete data elements in the EMR may not be the same across EMR software packages from different vendors. For instance, highly flexible, free-text-based EMR systems allow for multiple ways to enter data, requiring the identification of all options for denoting presence of disease conditions.

It is possible that we did not capture all typographical errors for prescription information or all the different naming conventions for the same laboratory test. Therefore, our results might have underestimated EMR capture. Practice Solutions EMR software is very free-text based and has good searching capabilities; thus, physicians tend to scan external documents such as diagnostic tests and consultation letters in an optical character recognition format to allow the text within those documents to be searched. In addition, types of external documents are classified as a consultation letter from a particular type of specialist and can be found under that specific variable (ie, consultation letter from a cardiologist). Not all EMR software is structured in this way, so some EMRs’ external documents are not classified and external documents are entered in PDF or TIFF format, which is a picture-like format that cannot be searched or de-identified. Thus, the detailed information that can be found in these documents may not be accessible.

Although highly promising,¹⁵ using EMR data for secondary purposes is not straightforward.^16,17 Full and accurate data availability may be possible in the case of integrated systems with health information technology that allows for a seamless, shared electronic record (as in Kaiser Permanente and the Veterans Health Administration).¹⁸ Policies and programs developed to improve seamless communication in large geographic areas between primary care EMRs and other parts of the healthcare system are needed. These automated electronic communication channels would further enhance the comprehensiveness of information captured within the EMR, which could lead to more efficient and informed patient care as well as improvement in the quality of the information for research and analytic purposes. Canada has no practices or polices in place for standardizing data entry into EMRs, and with limited pay-for-performance programs that would incentivize physicians to accurately code diagnostic information in structured data fields, analyzing EMR data will continue to have challenges.

Despite these limitations of EMR data for secondary purposes, the potential benefits include the ability to objectively study processes and outcomes of care, use the EMR data as a reference standard for administrative data validation, perform quality-of-care measures, give performance feedback to physicians, and efficiently identify cohorts of patients for clinical trials or other analyses. Important benefits of the combination of administrative data with EMR data are the combination of both financial and clinical data, and the ability to assess outcomes and outcomes per healthcare dollar spent.

A broader view of individual patients will include behavioral, environmental, and genomics data. However, in the absence of a well-established integrated system, use of EMR data that are linked with administrative data (as in EMRALD) offers an important first step toward providing a comprehensive picture of patient health histories across the spectrum of care.Author Affiliations: All authors-Institute for Clinical Evaluative Sciences (ICES) Toronto, Canada KT, NI, LJ-Department of Family and Community Medicine-University of Toronto, Toronto, Canada KT-Toronto Western Hospital Family Health Team-University Health Network, Toronto, Canada, NI-Department of Family Medicine-Women’s College Hospital, Toronto, CanadaLJ-Department of Family Medicine-Sunnybrook Health Sciences Centre, Toronto, CanadaDL-Department of Medicine-University Health Network, University of Toronto, Toronto, Canada, JT-Sunnybrook Shulich Heart Centre, University of Toronto, Toronto, Canada.

Funding Source: This work was funded by a Canadian Institutes of Health Research Team Grant in Cardiovascular Outcomes Research to the Canadian Cardiovascular Outcomes Research Team. This study was supported by the Institute for Clinical Evaluative Sciences (ICES), which is funded by an annual grant from the Ontario Ministry of Health and Long-Term Care (MOHLTC). The opinions, results, and conclusions reported in this study are those of the authors and are independent from the funding sources. No endorsement by ICES or the Ontario MOHLTC is intended or should be inferred.

Author Disclosures: Dr. Karen Tu is supported by a Canadian Institutes of Health Research Fellowship Award in Primary Care.Dr. Douglas Lee is a Clinician Scientist of the Canadian Institutes of Health Research.Dr. Jack Tu is supported by a Canada Research Chair in Health Services Research and a Career Investigator award from the Heart and Stroke Foundation of Ontario. Dr. Noah Ivers is supported by a Canadian Institutes of Health Research Fellowship Award in Clinical Research and by a Fellowship Award from the Department of Family and Community Medicine, University of Toronto-Doug Kavanagh was a System Architecture Consultant for Practice Solutions Software otherwise all other authors have no relevant conflicts of interest to disclose.

Authorship Information: Concept and design (KT, TFM, DL, JVT); acquisition of data (KT, DGK); analysis and interpretation of data (KT, TFM, NI, HL, HG, LJ, DL, JVT); drafting of the manuscript (KT); critical revision of the manuscript for important intellectual content (KT, TFM, NI, HL, HG, LJ, DGK, DL, JVT); statistical analysis (HL, HG); obtaining funding (KT, DL, JVT); and supervision (KT).

Address correspondence to: Karen Tu, Senior Scientist, Associate Professor, Family Physician. ICES, G1 06, 2075 Bayview Avenue, Toronto, Ontario M4N 3M5. E-mail: karen.tu@ices.on.ca1. Schoen C, Osborn R, Doty MM, Squires D, Peugh J, Applebaum S. A survey of primary care physicians in eleven countries, 2009: perspectives on care, costs, and experiences. Health Aff (Millwood). 2009;28(6): w1171-w1183.

2. Blumenthal D, Tavenner M. The “meaningful use” regulation for electronic health records. N Engl J Med. 2010;363(6):501-504.

3. Canada Health Infoway. 2015: advancing Canada’s next generation of healthcare. https://www.infoway-inforoute.ca/. Accessed June 4, 2012.

4. Schoen C, Osborn R, Squires D, et al. A survey of primary care doctors in ten countries shows progress in use of health information technology, less in other areas. Health Aff (Millwood). 2012;31(12): 2805-2816.

5. Goel V, Williams J, Anderson G, Blackstien-Hirsch P, Fooks C, Naylor D. A summary of studies on the quality of health care administrative databases in Canada. In: Patterns of Health Care in Ontario. The ICES Practice Atlas. 2nd ed. Ottawa: Canadian Medical Association; 1996:339-345.

6. Devoe JE, Gold R, McIntire P, Puro J, Chauvie S, Gallia CA. Electronic health records vs Medicaid claims: completeness of diabetes preventive care data in community health centers. Ann Fam Med. 2011; 9(4): 351-358.

7. Institute for Clinical Evaluative Sciences. Privacy Code: Protecting Personal Health Information at ICES. http://www.ices.on.ca/file/ICES_Privacy_Code_July_2011_v9new.pdf. Published July 2011. Accessed June 4, 2012.

8. OntarioMD Funding Eligible CMS Offerings/EMR Advisor. https://www.emradvisor.ca/compare. Accessed June 4, 2012.

9. Glazier RH, Zagorski BM, Rayner J. Comparison of Primary Care Models in Ontario by Demographics, Case Mix and Emergency Department Use, 2008/09 to 2009/10. Toronto, Canada: Institute for Clinical Evaluative Sciences; 2012.

10. IMS Health Canada. Pharmaceutical trends: top 50 dispensed medications in Canada, 2008. http://www.imshealth.com/deployedfiles/imshealth/Global/Americas/North%20America/Canada/StaticFile/Trends03_En09.pdf. Published 2009. Accessed June 4, 2012.

11. Tang PC, Ralston M, Arrigotti MF, Qureshi L, Graham J. Comparison of methodologies for calculating quality measures based on administrative data versus clinical data from an electronic health record system: implications for performance measures. J Am Med Inform Assoc. 2007;14(1):10-15.

12. Jackevicius CA, Li P, Tu JV. Prevalence, predictors, and outcomes of primary nonadherence after acute myocardial infarction. Circulation. 2008;117(8):1028-1036.

13. Kripalani S, LeFevre F, Phillips CO, Williams MV, Basaviah P, Baker DW. Deficits in communication and information transfer between hospital-based and primary care physicians: implications for patient safety and continuity of care. JAMA. 2007;297(8):831-841.

14. Chan KS, Fowles JB, Weiner JP. Review: electronic health records and the reliability and validity of quality measures: a review of the literature. Med Care Res Rev. 2010;67(5):503-527.

15. Balfour DC 3rd, Evans S, Januska J, et al. Health information technology—results from a roundtable discussion. J Manag Care Pharm. 2009;15(1)(suppl A):10-17.

16. Roth CP, Lim YW, Pevnick JM, Asch SM, McGlynn EA. The challenge of measuring quality of care from the electronic health record. Am J Med Qual. 2009;24(5):385-394.

17. Rea S, Pathak J, Savova G, et al. Building a robust, scalable and standards-driven infrastructure for secondary use of EHR data: the SHARPn project. J Biomed Inform. 2012;45(4):763-771.

18. Protti D. Integrated care needs integrated information management and technology. Healthc Q. 2009;13(spec No.):24-2