The Value of Health Information Technology: Filling the Knowledge Gap

ABSTRACT

Despite rapid growth in the rate of adoption of health information technology (HIT), and in the volume of evaluation studies, the existing knowledge base for the value of HIT is not advancing at a similar rate. Most evaluation articles are limited in that they use incomplete measures of value and fail to report the important contextual and implementation characteristics that would allow for an adequate understanding of how the study results were achieved. To address these deficiencies, we present a conceptual framework for measuring HIT value and we propose a checklist of characteristics that should be considered in HIT evaluation studies. The framework consists of 3 key principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used. Through examples, we show how these principles can be used to guide and improve HIT evaluation studies. The checklist includes a list of contextual and implementation characteristics that are important for interpretation of results. These improvements will make future studies more useful for policy makers and more relevant to the current needs of the healthcare system.

Am J Manag Care. 2014;20(11 Spec No. 17):eSP1-eSP8

Healthcare organizations all over the United States are adopting health information technology (HIT), yet recent studies of HIT are often unhelpful to these organizations or to those making policy decisions. Most studies fail to use adequate measures of value or to report key contextual and implementation characteristics. In this commentary, we:

• Demonstrate the problem in a sampling of published articles.
• Propose a conceptual framework and principles for measuring value in HIT.
• Present a checklist of key study characteristics.

Readers can use this article as a touchstone for assessing the utility and relevance of published research studies.In 2010, the United States Congress enacted the Health Information Technology for Economic and Clinical Health (HITECH) Act, thereby dedicating roughly $27 billion to eligible healthcare providers who demonstrate “meaningful use” of health information technology (HIT). This action motivated the purchase and installation of tens of thousands of HIT products across the US healthcare system and placed a national spotlight on HIT. The HITECH Act was justified, in part, by early studies that documented positive effects on healthcare quality, safety, and efficiency at a subset of institutions that were early to implement HIT, and by policy analyses that extrapolated the implications of these potential benefits throughout the US healthcare system.^1,2 These studies suggested that HIT had the potential to create enormous value by facilitating improvements in the quality of healthcare and reductions in costs.

However, systematic reviews of HIT have found that the evidence for value is inconclusive and that existing studies suffer from major limitations.^3-6 This finding is true even of the most recent literature reviews, despite a greatly increasing number of studies evaluating HIT.⁶ In this paper, we suggest a way to overcome these deficiencies and make HIT evaluations more relevant to the current needs of the healthcare system, by presenting a conceptual framework for measuring the value of HIT, examining how a sample of published HIT articles report the information needed to make meaningful assessments of value, and proposing a set of criteria for future evaluations that would make them more useful for policy makers.

This paper was sponsored by the Office of the National Coordinator for Health Information Technology, which otherwise had no role in the conduct or writing of the paper or the decision to publish it.

MEASURING THE VALUE OF HIT

Published studies have evaluated HIT using many different measures, including cost savings,⁷ time savings,^8,9 utilization reduction,^10-12 process adherence improvements,^13,14 health outcomes,^15,16 safety improvements,^12,17 physician satisfaction,¹⁸ patient satisfaction,¹⁸ and many others. These outcomes are all components of value, but all are limited. It would be a mistake to make a general conclusion about the value of HIT by looking at only 1 of these measures. To make more general assessments, evaluations of HIT should take into account 3 principles: 1) value includes both costs and benefits; 2) value accrues over time; and 3) value depends on which stakeholder’s perspective is used.

Principle 1: Value Includes Both Costs and Benefits

Value, by definition, includes both costs and benefits. Studies that report only costs (or proxies for cost, such as utilization) cannot be used to reach conclusions about value. For example, 1 study quantified the costs of implementing an electronic health record (EHR) system for a 5-physician practice, including direct financial expenditures and hours needed to implement.¹⁹ These estimates of cost are an important component of value to that physician practice, but cannot by themselves be considered an estimate of value, because they do not include an assessment of potential benefits, which could include reduced costs that may accrue downstream. Nor do they include health benefits to patients, which may accrue in the near or long term as a result of the HIT implementation.

In another example of the limited ability of costs-only studies to define value, a population-based clinical decision support system was shown to increase the use of outpatient services and total medical costs in the intervention group, whereas services and costs did not significantly change in the control group.²⁰ Again, without an assessment of benefits, these results are not a complete assessment of value, and general conclusions should not be made.

As a third example, the United States as a whole spends far more on healthcare now than it did in 1950, in large part due to technological advances. A newborn in 1950 could expect to require $8000 worth of medical care over the course of his or her lifetime (in 2001 dollars discounted to birth), whereas the comparable figure for a newborn in 1990 was $45,000, a nearly 600% increase. Yet, as Cutler and McClellan argue, based on an analysis of 5 medical conditions (heart attacks, low birth weight babies, depression, cataracts, and breast cancer), despite the rapid rise in costs, the benefits of medical progress (in terms of quality-adjusted life years) more than justify the added costs ($7 in benefits for every $1 in increased spending for heart attack treatments, for example).²¹

While it may not be feasible to measure all the costs and all the benefits of a particular HIT functionality, studies that only measure one or the other cannot by themselves provide assessments about the value of the technology.

Principle 2: Value Accrues Over Time

Value accrues in various degrees over time. The costs of an EHR, for example, from the perspective of a medical group, may entail purchase costs for the hardware and software, installation expenses, and training fees. These are all short-term costs. In the longer term, the EHR may require varying levels of maintenance, replacement costs, and retraining. Costs in terms of time burden may also vary over time, as clinicians become more adept at using the user interface.

Benefits vary across time as well. From the perspective of a patient, in the near term, HIT may result in a reduction in errors. In the long term, HIT may facilitate many health benefits through clinical monitoring, reminders, and disease management and prevention. In many cases, HIT may have both short- and long- term benefits. A computerized alert prompting the measurement of fasting lipids in a 40-year-old male may set the patient on a healthier trajectory, which may accrue over decades in cardiovascular disease avoided.

It is not feasible for any particular study to capture all the effects (especially in the long term) of any particular HIT intervention. However, the time horizon should be long enough to reflect the differences between the benefits and costs of using the HIT intervention versus not using it.²²

Principle 3: Value Depends on Which Stakeholder’s Perspective Is Used

There are many possible perspectives from which to view the value of HIT. For example, consider 2 physician group practices, located across the street from each other,

both of which have implemented the same HIT system. Through the use of physician-reminder prompts, the delivery of screening colonoscopies increased 2-fold among eligible patients for both practices. If one physician group practice were reimbursed under fee-for-service, the increase in services would be a source of revenue, and therefore an increase in value to the practice. If instead of fee-for-service the other physician group practice were paid by capitation, this increase in services would be a cost to the group (in the short term, at least) and therefore a decrease in value. Even if the benefits to patients were equal in the 2 practices, the perception of value to the practices would be very different.

As another example, from the perspective of a small provider operating under fee-for-service, an EHR may necessitate that the physicians see fewer patients because of the increased documentation burden. But from a patient’s perspective, the improved documentation may result in better clinical decision making and improved health outcomes. From another patient’s perspective, however, the decrease in visit volume may have a negative effect in that it may make it more difficult for him or her to book a visit. From the payer’s perspective in this example, there may be a decrease in charges from the provider’s office for a period of time during the transition to an EHR, saving the payer money. However, more expensive emergency care may end up substituting for some of those visits, in which case the costs may increase, at least in the short term.

Thus, in any application of HIT, there may be “winners” and “losers” in terms of certain aspects of costs or utilization. In the Veterans Health Administration (VHA), for example, it takes more time for the physicians to enter their progress notes into the electronic health record than it did when paper was used, so the physicians “lose” in terms of time. But the VHA “wins” via reductions in the costs of storing and retrieving all the paper-based records, and the patients “win” by getting more preventive care services delivered to them, because the EHR contains decision support that prompts providers to deliver these services to eligible people. In some situations, the “winner” and “loser” is the same entity, depending on what is being looked at: physicians “lose” because it takes more time to enter information into the EHR, but they also “win” because every other physician has also had to enter their notes into the EHR, which means that all the time lost trying to figure out what happened at the last hospitalization or visit to the subspecialist disappears when this information is instantly available in the same EHR. These kinds of “transfers” can vastly influence conclusions about value, depending on the perspective.

Only 1 perspective is insensitive to such transfers when assessing the value of HIT, and it can be used as a summative evaluation for HIT as a whole: social value. Social value adds up all the costs paid directly or indirectly by patients (taxes, premiums through employers, premiums directly paid, direct payments as co-pays or self-pay) and benefits to patients (convenient access to providers, better health, longer lives). Any complete assessment of value of HIT for society must be from this social perspective.²³

Social value does not always align with the business value to providers. If a physician group or hospital operates more efficiently and saves money as a result of HIT but the quality of the healthcare it delivers is unchanged, the social value of healthcare is also unchanged. However, if market forces result in healthcare organizations lowering the costs they charge payers, and if the payer passes on those savings to patients through decreased premiums, the social value will increase as a result of HIT. Unfortunately, market forces are imperfect in healthcare, and cost savings often do not get passed back to the patient, or if they do, it takes many years. It is therefore important for studies of HIT value to state how they expect the market to respond to the effect of the technology, including the time frame in which these changes may be reasonably expected to occur.

Ideally, all evaluations of HIT would take the perspective of all relevant stakeholders, including all possible costs and benefits, and consider the downstream effects of the technology in future decades. Clearly, such studies would be infeasible. However, we believe that authors of studies could do a better job in specifying which stakeholder’s perspective they are using as the lens for their evaluation, and in explaining the link to social value.

WHAT INFORMATION ABOUT VALUE IS CURRENTLY REPORTED IN STUDIES OF HIT?

Table 1

Unfortunately, we have found that few studies include both costs and benefits in their definitions of value. Most studies look at only short-term time horizons, which ignore many of the downstream benefits of the HIT, and many studies don’t even explicitly state to whom the value is accruing. To illustrate this, we present in a sample of recent HIT articles, and whether and how they reported information on value (columns A through D). These articles are a purposive sample to capture a diversity of study types, including those that focus on 1 particular HIT installation as well as broader cross-sectional studies.

In this sample of 10 articles, 3 reported only on costs, without mention of any benefits, while 1 reported only on benefits, with no reporting on costs (which can include utilization). Thus, 4 of the 10 studies do not report both costs and benefits, and therefore cannot provide information about value (since value includes both costs and benefits). All of the studies are relatively short term, most having a time horizon of 1 to 3 years, and thus any downstream benefits or costs that might accrue due to better care today will not be captured. Only 1 study was explicit about “To whom does the value accrue?”, although in 3 others this could be inferred. These studies clearly show room for improvement in terms of how they measure value.

Even when information on costs, benefits, and perspective are presented, studies of HIT must then present sufficient information for readers to know what the intervention is, if readers are to be able to reach conclusions about whether the HIT system is something that could be implemented in another context. However, most evaluations of HIT do not adequately report such information. Few studies explain how the technology works, let alone how it was implemented. Furthermore, most studies don’t even report basic contextual information such as under what payment system the users were paid.

Studies need to open up the “black box” of HIT and do a better job explaining how HIT contributed to the results, including the role of key contextual factors. The need for particulars about the intervention details and implementation details has been noted by others for several years.^4,34,35 All of these authorities recommend including at least 3 descriptors of the intervention: what the functionality does, contextual information, and implementation information. HIT evaluation studies need to describe this information so that a reader can understand what was being evaluated and assess whether or not they might be able to realize that value if they were to try to implement the same intervention in a different context.

To show how studies currently achieve this benchmark, we assessed the same 10 studies as above along these dimensions (Table 1, columns E, F, and G). All the studies had some description of the HIT functionalities, although for some this was at a very general and basic level (“a hospital EHR with CPOE [computerized physician order entry],” otherwise unspecified). Four studies report no contextual information at all, while in another 3, the only contextual data are the size of the organizations (usually a hospital). Only 3 studies reported some aspect of the financial status or payer mix. In 8 studies, there was no information about implementation. These data support, broadly, conclusions similar to those reached in a recent systematic review of HIT covering the years 2010-2013. In that review of 225 studies, only about a third of studies reported any information about context and implementation beyond the simplest measures.⁶

A PROPOSAL TO IMPROVE FUTURE HIT EVALUATIONS OF VALUE

Table 2

Without more attention to the necessary measuring and reporting of the data needed to assess value, we risk the possibility of 3 more years’ worth of published studies, which we estimate would be more than 300 hypothesis-testing articles,6 that do not give us appreciably better knowledge about this crucial aspect of HIT: how best to realize value. We propose in criteria that should be met by all reports of HIT studies that are intended to improve our understanding of value. These include using evaluations to include both costs and benefits, using longer time horizons, considering the perspectives of different stakeholders, and reporting implementation details and contextual variables.

Making these changes may require especially imaginative study designs, and research teams that have both quantitative and qualitative expertise. For example, more qualitative research will be required to assess such questions as: How were users of the HIT trained? Was the training likely effective? What kind of support was provided from organizational leadership? What exactly was the HIT product intended to do? To what extent were the end users involved in the design of the technology? Details like these are critical to determining the applicability of any particular study’s results to other settings. While an extensive body of qualitative literature shows how HIT may achieve benefits or result in unintended consequences, this literature is almost completely distinct from studies that assess value in terms of quantitative measures of efficiency, quality, and safety. It is therefore largely impossible, using the current literature, to both identify the successful implementations of HIT and understand how to replicate them.

Making such changes will not be easy, as ideas about what constitutes HIT research are entrenched. However, if research is to align with the current needs of the healthcare system, researchers should take on this challenge and produce results that not only prove what is possible, but also show how to achieve it.Author Affiliations: RAND Health, Santa Monica, CA (PS, RJH, EBK) and Boston, MA (RSR, SSJ); Harvard Medical School, Division of General Internal Medicine, Brigham and Women’s Hospital (SSJ), Boston, MA; West Los Angeles VA Medical Center (PS), Los Angeles, CA; Vanguard Health Systems (SSJ), Nashville, TN.

Source of Funding: This work was funded by the Office of the National Coordinator for Health Information Technology under contract number HHSP23320095649WC.

Author Disclosure: Dr Shekelle is paid royalties from UpToDate, is a member of the National Guideline Clearinghouse editorial board, and is employed by the Department of Veterans Affairs. Dr Jones is now employed by Tenet Healthcare, which is in a contract relationship with Cerner Corp, which provides the organization's electronic health record system. Drs Rudin, Keeler, and Hillestad report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (RSR, SSJ, PS, RJH, EBK); acquisition of data (SSJ, PS); analysis and interpretation of data (SSJ, PS); drafting of the manuscript (RSR, SSJ, PS, RJH); critical revision of the manuscript for important intellectual content (RSR, SSJ, PS, RJH, EBK); obtaining funding (SSJ, PS); supervision (RSR, PS).

Address correspondence to: Robert S. Rudin, PhD, RAND Health, 20 Park Plaza, Ste 920, Boston, MA 02116. E-mail: rrudin@rand.org.REFERENCES

1. Hillestad R, Bigelow J, Bower A, et al. Can electronic medical record systems transform health care? potential health benefits, savings, and costs. Health Aff (Millwood). 2005;24(5):1103-1117.

2. Walker J, Pan E, Johnston D, Adler-Milstein J, Bates DW, Middleton B. The value of health care information exchange and interoperability. Health Aff (Millwood). 2005;Suppl Web Exclusives:W5-10-W5-8.

3. Buntin MB, Burke MF, Hoaglin MC, Blumenthal D. The benefits of health information technology: a review of the recent literature shows predominantly positive results. Health Aff (Millwood). 2011;30(3):464-471.

4. Chaudhry B, Wang J, Wu S, et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006;144(10):742-752.

5. Goldzweig CL, Towfigh A, Maglione M, Shekelle PG. Costs and benefits of health information technology: new trends from the literature. Health Aff (Millwood). 2009;28(2):w282-w293.

6. Shekelle PG, Jones SS, Rudin RS, et al, of the Southern California Evidence-based Practice Center of the RAND Corporation. Health Information Technology: An Updated Systematic Review with a Focus on Meaningful Use Functionalities. Washington, DC: Office of the National Coordinator for Health Information Technology; 2014.

7. Pettit RS, Zimmerman CR, Alaniz C, Dorsch MP. Cost analysis before and after implementation of a computerized physician order entry order form for enoxaparin. PT. 2012;37(2):107-111.

8. Devine EB, Hollingworth W, Hansen RN, et al. Electronic prescribing at the point of care: a time-motion study in the primary care setting. Health Serv Res. 2010;45(1):152-171.

9. Duffy WJ, Kharasch MS, Du H. Point of care documentation impact on the nurse-patient interaction. Nurs Adm Q. 2010;34(1):E1-E10.

10. Furukawa MF. Electronic medical records and efficiency and productivity during office visits. Am J Manag Care. 2011;17(4):296-303.

11. Mäenpää T, Asikainen P, Gissler M, et al. Outcomes assessment of the regional health information exchange: a five-year follow-up study. Methods Inf Med. 2011;50(4):308-318.

12. Stenner SP, Chen Q, Johnson KB. Impact of generic substitution decision support on electronic prescribing behavior. J Am Med Inform Assoc. 2010;17(6):681-688.

13. Bell DS, Cima L, Seiden DS, Nakazono TT, Alcouloumre MS, Cunningham WE. Effects of laboratory data exchange in the care of patients with HIV. Int J Med Inform. 2012;81(10):e74-e82.

14. Herrin J, da Graca B, Nicewander D, et al. The effectiveness of implementing an electronic health record on diabetes care and outcomes. Health Serv Res. 2012;47(4):1522-1540.

15. Quinn CC, Shardell MD, Terrin ML, Barr EA, Ballew SH, Gruber-Baldini AL. Cluster-randomized trial of a mobile phone personalized behavioral intervention for blood glucose control. Diabetes Care. 2011;34(9):1934-1942.

16. Gustafson D, Wise M, Bhattacharya A, et al. The effects of combining Web-based eHealth with telephone nurse case management for pediatric asthma control: a randomized controlled trial. J Med Internet Res. 2012;14(4):e101.

17. Abramson EL, Barrón Y, Quaresimo J, Kaushal R. Electronic prescribing within an electronic health record reduces ambulatory prescribing errors. Jt Comm J Qual Patient Saf. 2011;37(10):470-478.

18. Duffy RL, Yiu SS, Molokhia E, Walker R, Perkins RA. Effects of electronic prescribing on the clinical practice of a family medicine residency. Fam Med. 2010;42(5):358-363.

19. Fleming NS, Culler SD, McCorkle R, Becker ER, Ballard DJ. The financial and nonfinancial costs of implementing electronic health records in primary care practices. Health Aff (Millwood). 2011;30(3):481-489.

20. Eisenstein EL, Anstrom KJ, Edwards R, Willis JM, Simo J, Lobach DF. Population-based clinical decision support: a clinical and economic evaluation. Stud Health Technol Inform. 2012;180:343-347.

21. Cutler DM, McClellan M. Is technological change in medicine worth it? Health Affairs. 2001;20(5):11-29.

22. Weinstein MC, O’Brien B, Hornberger J, et al; ISPOR Task Force on Good Research Practices—Modeling Studies. Principles of good practice for decision analytic modeling in health-care evaluation: report of the ISPOR Task Force on Good Research Practices—Modeling Studies. Value Health. 2003;6(1):9-17.

23. Gold M. Panel on cost-effectiveness in health and medicine. Med Care. 1996;34(12 suppl):DS197-DS199.

24. Connelly DP, Park YT, Du J, et al. The impact of electronic health records on care of heart failure patients in the emergency room. J Am Med Inform Assoc. 2012;19(3):334-340.

25. Furukawa MF, Raghu TS, Shao BB. Electronic medical records, nurse staffing, and nurse-sensitive patient outcomes: evidence from California hospitals, 1998-2007. Health Serv Res. 2010;45(4):941-962.

26. Himmelstein DU, Wright A, Woolhandler S. Hospital computing and the costs and quality of care: a national study. Am J Med. 2010;123(1):40-46.

27. Lau AY, Sintchenko V, Crimmins J, Magrabi F, Gallego B, Coiera E. Impact of a web-based personally controlled health management system on influenza vaccination and health services utilization rates: a randomized controlled trial. J Am Med Inform Assoc. 2012;19(5):719-727.

28. Ling SB, Richardson DB, Mettenbrink CJ, et al. Evaluating a web-based test results system at an urban STI clinic. Sex Transm Dis. 2010; 37(4):259-263.

29. McCormick D, Bor DH, Woolhandler S, Himmelstein DU. Giving office-based physicians electronic access to patients’ prior imaging and lab results did not deter ordering of tests. Health Aff (Millwood). 2012;31(3):488-496.

30. Palen TE, Ross C, Powers JD, Xu S. Association of online patient access to clinicians and medical records with use of clinical services. JAMA. 2012;308(19):2012-2019.

31. Shapiro JS, Baumlin KM, Chawla N, et al. Emergency department information system implementation and process redesign result in rapid and sustained financial enhancement at a large academic center. Acad Emerg Med. 2010;17(5):527-535.

32. Virga PH, Jin B, Thomas J, Virodov S. Electronic health information technology as a tool for improving quality of care and health outcomes for HIV/AIDS patients. Int J Med Inform. 2012;81(10):e39-e45.

33. Zlabek JA, Wickus JW, Mathiason MA. Early cost and safety benefits of an inpatient electronic health record. J Am Med Inform Assoc. 2011;18(2):169-172.

34. Wears RL, Berg M. Computer technology and clinical work: still waiting for Godot. JAMA. 2005;293(10):1261-1263.

35. Talmon J, Ammenwerth E, Brender J, de Keizer N, Nykänen P, Rigby M. STARE-HI—statement on reporting of evaluation studies in Health Informatics. Int J Med Inform. 2009;78(1):1-9.

Download PDF: The Value of Health Information Technology: Filling the Knowledge Gap