Economics of a Health System’s Direct-to-Consumer Telemedicine for Its Employees

ABSTRACT

Objectives: To compare the mean per-episode unit cost for a direct-to-consumer (DTC) telemedicine service for medical center employees (OnDemand) with that of in-person care and to estimate whether the offered service increased the use of care.

Study Design: Propensity score–matched retrospective cohort study of adult employees and dependents of a large academic health system between July 7, 2017, and December 31, 2019.

Methods: To estimate differences in per-episode unit costs within 7 days, we compared costs between OnDemand encounters and conventional in-person encounters (primary care, urgent care, and emergency department) for any similar condition using a generalized linear model. We used interrupted time series analyses limited to the top 10 clinical conditions managed by OnDemand to estimate the effect of OnDemand’s availability on the trends for overall employee per-month encounters.

Results: A total of 10,826 encounters among 7793 beneficiaries were included (mean [SD] age, 38.5 [10.9] years; 81.6% were women). The mean (SE) 7-day per-episode cost among employees and beneficiaries was lower for OnDemand encounters at $379.76 ($19.83) relative to non-OnDemand encounters at $493.49 ($25.53), a mean per-episode savings of $113.73 (95% CI, $50.36-$177.10; P < .001). After the introduction of OnDemand, among employees with encounters for the top 10 clinical conditions managed by OnDemand, the trend for encounter rates per 100 employees per month increased marginally (0.03; 95% CI, 0.00-0.05; P = .03).

Conclusions: These results suggest that DTC telemedicine staffed by an academic health system and offered directly to employees reduced the per-episode unit costs and only marginally increased utilization, suggesting lower cost overall.

Am J Manag Care. 2023;29(6):284-290. https://doi.org/10.37765/ajmc.2023.89369

Takeaway Points

The value of direct-to-consumer (DTC) telemedicine services offered by academic health systems is understudied.

• DTC telemedicine services for low-acuity or minor illnesses are increasingly offered as an employee benefit, but any per-episode unit cost advantage may be offset by overuse of care.
• DTC telemedicine staffed by an academic health system and offered to its employees resulted in lower per-episode unit costs for care within 7 days and only marginally increased the use of services.
• DTC telemedicine staffed by an academic health system and offered directly to employees was cost-saving.

Employers in the United States have increasingly been offering a direct-to-consumer (DTC) telemedicine benefit for low-acuity or minor illnesses to their employees.^1-3 By 2021, more than 95% of employers with 50 or more employees provided some coverage for DTC telemedicine in their largest health plan; more than 75% felt that offering telemedicine was important and nearly 20% either limited or eliminated cost sharing for telemedicine.⁴

Despite these trends among general employers, few health systems have directly provided DTC telemedicine to their own employees. An evaluation of employer-supported DTC telemedicine suggests that it does not lower costs.⁵ However, unlike the general case of DTC telemedicine staffed by third-party firms with clinicians stationed throughout the United States, a local health system’s DTC telemedicine option could offer efficiencies: It can be staffed by the system’s own clinicians who are familiar with the nearby care delivery environment, use the health system’s electronic health record, and refer and coordinate follow-up in-person care within the system, minimizing care fragmentation or out-of-system care by its employees. An equivocal case for DTC telemedicine for a conventional employer may be a much clearer win when the employer is also a health system. Additionally, with the increased telemedicine capacity and capability of health systems since 2020, the possibility of providing DTC telemedicine to their employees has increased.

The potential cost savings from DTC telemedicine derive from the lower overhead and operational costs typically required by these services, which can be reflected in a lower price per unit of service. But because these services are easy to access (often available immediately, around the clock, and without travel), they may induce overuse of care, especially for self-limited conditions such as viral upper respiratory infections for which the alternative to in-person care is no care at all, thus increasing the overall cost of care.^5-11 Telemedicine will save money relative to in-person care if any unit price advantages are not overwhelmed by the increased use of care overall, induced by its convenience.

Much is at stake. Employers provide health insurance coverage for 158 million Americans or nearly 50% of the population. Since the COVID-19 pandemic began, telemedicine has represented a significantly larger portion of all medical claims—consistently more than 5% of all medical claims by mid-2021^12-15—and the estimated value of the global telemedicine industry is projected to reach a quarter of a trillion dollars by 2024.¹³ Yet, the future of telemedicine remains undetermined with reimbursement rates in debate,^16-18 driven in large part because its economic value is understudied and uncertain.

We conducted an economic evaluation of DTC telemedicine owned, operated, and offered by a large academic health system to its employees from the time of its initial offering on July 1, 2017, until December 31, 2019. We estimated and compared the mean per-episode unit cost of telemedicine visits provided by a health system with that of in-person care for the same set of conditions managed using DTC telemedicine. Then, we estimated whether telemedicine’s availability influenced overall care utilization trends and cost after it was introduced as an employee benefit.

METHODS

This study was deemed exempt by the institutional review board at the University of Pennsylvania. We followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guideline for observational studies.

Setting

Penn Medicine is an academic health system serving large portions of southeast Pennsylvania, New Jersey, and Delaware. Penn Medicine is self-insured and more than 95% of employees use its only employer-sponsored plan—a preferred provider organization (PPO) plan—rather than insurance obtained individually or through a family member. Since 2017, these PPO-insured employees have been offered Penn Medicine OnDemand,¹⁹ a 24/7 DTC telemedicine benefit to employees and their adult (≥ 18 years) dependents. The DTC telemedicine encounter requires no co-payment or other cost sharing, but subsequent laboratory, imaging, or prescription fills may result in standard out-of-pocket costs. The PPO plan routinely covers more than 38,000 beneficiaries consisting of employees and dependents.

OnDemand was staffed by Penn Medicine–employed nurse practitioners who conducted live telemedicine appointments from a central technology hub. The nurse practitioners were overseen by a medical director (K.H.C.) and supported by onsite 24/7 support and administrative staff who coordinated follow-up care for patients within Penn Medicine’s network of clinical practices or suggested nearby non–Penn Medicine health care locations (eg, urgent care centers, emergency departments [EDs]), when subsequent in-person care was determined necessary.

Data Sources

To measure utilization and calculate the total cost of care from the perspective of the health system as both payer and provider, we used 2016-2019 claims datafor Penn Medicine PPO’s adult (≥ 18 years) beneficiaries. The data file included employee age, gender, zip code, health care utilization within and outside Penn Medicine by the site of outpatient ambulatory care (EDs, urgent care centers, retail health clinics, physician offices, laboratories, imaging centers, and pharmacies), and the amount paid by the plan. OnDemand encounter dates and the primary diagnosis for each visit were extracted from the health system’s electronic health record and merged with the plan’s claims data. The data linkage between employee claims and the electronic health record was approved by Penn Medicine’s Employee Health Privacy Board after applying the Health Insurance Portability and Accountability Act’s minimum necessary requirement.

Study Sample

We measured health care use between July 1, 2017, and December 31, 2019. To provide insights into DTC telemedicine’s future economic impact, we excluded 2020 and subsequent data during the pandemic because health care utilization was sensitive to surges during the pandemic,^20-22 which may be less reflective of future nonpandemic utilization, a goal of our study.

Per-Episode Costs

We included only beneficiaries who were continuously enrolled in the PPO plan for 12 months prior to the index OnDemand or non-OnDemand visit, evidenced by at least 1 insurance claim (eg, outpatient, pharmacy, or laboratory claim) during the preceding year. We then matched index encounters from OnDemand encounters with index encounters among non-OnDemand users by diagnosis. For index encounters, we grouped the International Classification of Diseases, Tenth Revision (ICD-10) diagnosis using the Clinical Classification Software Refined (CCSR) from the Healthcare Cost and Utilization Project.²³ The CCSR is a uniform and standardized coding system that uses an ICD-10 diagnosis code and collapses the multitude of ICD codes into a smaller number of clinically meaningful categories. We limited the index encounters for non-OnDemand users to in-person visits to primary care, an ED, or urgent care facility for the same set of CCSR-categorized primary diagnoses used in OnDemand encounters. Outside of OnDemand, less than 0.01% of outpatient claims filed during the study period occurred via telemedicine.

OnDemand users could have multiple index OnDemand encounters in the data file. If multiple OnDemand encounters occurred within 30 days (13.5% of encounters), only the initial encounter was included in the data file for the purpose of defining an index encounter; the subsequent OnDemand encounter was captured in cost calculations.

Because the choice to use OnDemand or a conventional in-person visit is nonrandom, we used 1:1 propensity score matching for OnDemand (intervention) and conventional encounters (controls). This approach reduces selection bias by using observable characteristics to identify a control population that is similar to the intervention cohort.^24,25 Balance was considered to be achieved if the postmatching standardized mean difference (SMD) was less than 0.10.

Propensity scores were estimated using logistic regression based on the index encounter’s CCSR-categorized primary diagnosis, calendar year and quarter of the encounter, and beneficiary characteristics at the time of the index encounter: age, gender, employee or beneficiary status, Charlson Comorbidity Index score to adjust for beneficiary’s clinical risk factors,²⁶ median income of zip code of residence, prior costs (log-transformed to account for skewed cost data), and utilization in previous 12 months to reflect the need for care or tendency to receive care.^27,28

Using a payer perspective, per-episode cost was calculated using claims from the day of the index encounter through the next 6 days to capture 7 days of cost for both OnDemand and non-OnDemand cohorts. We used procedure codes and revenue codes (for hospital-based claims) to classify per-episode cost into the following categories: primary care, specialty care, ED, urgent care, other outpatient (eg, imaging, physical therapy), inpatient, other medical (eg, laboratory, durable medical equipment), and pharmacy. The health system’s cost of administering OnDemand (“operating costs”) was calculated separately because these costs were fixed, making the per-episode cost of administering OnDemand volume sensitive (more visits result in lower per-episode costs, and vice versa). Operating costs were calculated using the salaries and benefits for staff (eg, clinicians, support staff) and supplies (eg, computers, licensure). We estimated the per-episode unit cost of OnDemand encounters as $127.04 by dividing operating costs ($1,112,367) by the number of total encounters (8756) during the study period.

Substitution vs New Use

To estimate whether telemedicine encounters were substitutions that replaced in-person care or were additions to care that would otherwise not have occurred, we used interrupted time series analyses to estimate the effect of OnDemand’s availability on the trends for overall encounter rates by month, before and after the introduction of OnDemand. We used a data set that was modified from the data set for the per-episode unit cost analysis, expanding the data file to include January 1, 2016, to December 31, 2019 (a period before the introduction of OnDemand on July 1, 2017), and excluding dependents because of restrictions to their data file. We measured encounters for the top 10 CCSR-categorized primary diagnoses from OnDemand encounters (56% of all OnDemand claims), conditions for which the overall use of care would most likely have been changed by the availability of OnDemand. A significant difference between observed and expected use trends before and after OnDemand periods would suggest a change associated with the availability of OnDemand.

Statistical Analyses

For per-episode unit cost, we summed 7-day episode spending for each OnDemand and non-OnDemand encounter from our matched cohort using a generalized linear model specifying a gamma family, log link, and robust SEs to account for the skewed distribution of cost outcomes. The model accounted for individual-level clustering to account for multiple encounters by the same beneficiary. We conducted a sensitivity analysis by extending the follow-up window for per-episode unit cost to 30 days after the index encounter date. For the interrupted time series analyses, we used a segmented ordinary least squares regression that included month as a covariate to account for monthly variations in utilization. Statistical analyses were performed with SAS version 9.4 (SAS Institute).

RESULTS

Per-Episode Costs

After propensity score matching, 5413 OnDemand encounters among 3506 beneficiaries met inclusion criteria and were matched to 5413 non-OnDemand encounters among 4287 beneficiaries who had not used OnDemand but instead used in-person primary care, ED, or urgent care facility encounters for the same set of CCSR-categorized conditions. Table 1 shows differences between OnDemand and non-OnDemand beneficiary encounters before matching: OnDemand beneficiary encounters were more likely to involve beneficiaries who were young, female, employees (instead of spouses or dependents), and healthy, and these beneficiaries had a different distribution for the median household income of their zip code and care-seeking behaviors based on seasonality, prior year’s utilization, and prior year’s cost. After matching, these cohorts of encounters among OnDemand and non-OnDemand users were similar on all matching criteria (SMDs < 0.10). Table 2 shows the top 10 diagnoses assessed during an OnDemand encounter (56% of all diagnoses): sinusitis, upper respiratory tract infections, respiratory symptoms, urinary tract infections, eye symptoms, musculoskeletal symptoms, skin symptoms, allergies or allergic reactions, abdominal symptoms, and a variety of infections (Table 2).

Seven-day per-episode cost was lower for OnDemand users. Table 3 [A] compares the mean (SE) 7-day per episode cost of $379.76 ($19.83) for OnDemand users and $493.49 ($25.53) for non-OnDemand users, a mean per-episode savings of $113.73 (95% CI, $50.36-$177.10; P < .001). Expanding the observation window to 30 days for comparison in Table 3 [B], the total cost increased in both matched cohorts and the magnitude of savings from OnDemand was similar to differences observed with a 7-day window, but that difference was no longer statistically significant.

Substitution vs New Use

By the beginning of 2018, nearly 20% of all employee encounters for the top 10 CCSR-classified diagnoses occurred within OnDemand (Figure [A]). The trend in monthly encounters per 100 employees increased after the introduction of OnDemand by 0.03 encounters per 100 employees per month (95% CI, 0.00-0.05; P = .03) (Figure [B]). Over the study period, this difference in trend reflected a 9.7% increase in overall encounters for these conditions.

DISCUSSION

This study has 2 main findings. First, a telemedicine alternative to in-person care delivered within a medical center free to its employees produced lower immediate per-episode unit cost for care. This favorable per-episode cost might be undermined if OnDemand’s convenience induced care overuse for symptoms that otherwise would have been ignored by employees. A second finding is that encounter rates increased marginally after the introduction of OnDemand among a set of conditions for which inducement of overuse would have been most likely. Contextualizing these findings, the lower OnDemand unit cost of 77.0% of conventional care more than offset the 9.7% observed increase in services. Given that total cost is the product of price and volume, these findings suggest an estimated reduction in overall costs of approximately 15.6% during the study period.

Results from prior studies question the cost-savings potential of DTC telemedicine.²⁹ An evaluation of a national DTC telemedicine service offered to California’s public employees for low-acuity, urgent conditions observed lower per-episode unit cost for DTC telemedicine, but the total cost of care increased because the volume of encounters increased, suggesting substantial inducement of care overuse: low-value care seeking that might have been forgone had telemedicine not been offered.⁵ This analysis was limited to acute respiratory infections, and telemedicine was administered by a third-party telemedicine company. A study of telemedicine for low-acuity, urgent conditions offered by Intermountain Healthcare to its general patient population (ie, not limited to employees) found lower per-episode unit cost but did not evaluate substitution or inducement effects, therefore limiting our understanding of telemedicine’s impact on overall utilization and cost of care.³⁰ Our study differed from these prior studies because our economic evaluation of DTC telemedicine included all encounter diagnoses, we estimated telemedicine’s influence on overall utilization, and the DTC telemedicine service was operated by a large academic health system and offered to its own employees and their adult dependents.

Who operates, offers, staffs, and uses telemedicine may drive these favorable results. Third-party telemedicine companies are typically separate from local health systems, geographically and operationally. Partnerships with local health systems may offer advantages: (1) existing trust relationships may facilitate use; (2) follow-up care is less likely to involve service duplication, such as another in-person triage-type visit before reaching a more targeted appointment, because they are familiar with the local health care ecosystem; and (3) when employees already use that system for care, information in the records may facilitate the encounter and follow-up plan. Additional synergies emerge when the employer is also the provider of care: Any needed follow-up care might be directed in system (an absolute increase of 8% among OnDemand encounters in our study), where it is effectively purchased wholesale, rather than out of system, where it is effectively purchased at a retail price; and these types of benefits may improve retention of employees who feel cared for by their health care–providing employer. DTC third-party telemedicine companies often use physicians or a mixture of physicians and advanced practice providers. OnDemand was staffed by nurse practitioners only, limiting the overall costs of care. The design and financing of OnDemand also mattered. We suspect that the use of OnDemand would have been lower, and the outcomes might have been different had the health system imposed co-payments or cost sharing typical for conventional in-person care. Further, as a self-insured product, OnDemand operated in a non–fee-for-service arrangement, potentially incentivizing OnDemand clinicians to not be motivated by repeat visits or increased volume relative to a fee-for-service arrangement.

Limitations

This analysis has several limitations. First, we assessed cost in an employee population working within Penn Medicine’s large, regional academic health system. Penn Medicine employees, whether they used OnDemand or not, had a mean age in the mid-30s and they used the ED 10 times and urgent care 5 times more often than primary care for conditions commonly managed by OnDemand. Our observations may be reflective of telemedicine as a substitution for expensive in-person care tendencies of health care employees. Alternatively, health care personnel may use DTC telemedicine more efficiently, seeking care for conditions that may be more appropriate for telemedicine relative to in-person care. Second, the study population had a PPO insurance plan with generous benefits, including no co-pays for a telemedicine appointment with OnDemand. Individuals who are uninsured or have public insurance, a high-deductible health plan, or any insurance with a co-pay for a telemedicine appointment may use DTC telemedicine and health care differently. However, the marginal increase in utilization despite no co-pays for OnDemand is encouraging. Third, the future generalizability of our findings and observed utilization of OnDemand may change if patients have changed care-seeking behavior because of telemedicine’s increased availability or if telemedicine’s capabilities evolve (eg, increases in remote patient monitoring). Fourth, residual selection bias remains possible from unobserved characteristics influencing our observations. Fifth, to maintain employee privacy, we used a restricted set of sociodemographic and clinical data for matching; more granular data variables might have allowed for more precise matching or adjustments for uncaptured confounders. Sixth, although we do not capture or compare the quality of care or patient satisfaction with each care episode, our analyses indicate that few telemedicine encounters resulted in follow-up care in in-person settings for untreated or undertreated conditions. Seventh, since the onset of the COVID-19 pandemic in 2020, telemedicine has become more acceptable to patients, clinicians, and insurers. These changes will likely reduce the generalizability of these prepandemic findings but at the same time increase the relevance of the clinical model and economic modeling. Finally, a key potential advantage of telemedicine services is that employees can use them easily, typically without taking time off from work. The value of that convenience to employees and effects on work productivity measures were not captured.

Despite these limitations, this study has strengths. It reflects what is to our knowledge the largest and most comprehensive economic evaluation of a DTC telemedicine service, particularly among those offered by a health system. There is careful adjustment for differences in patient characteristics, which have a conservative bias.

CONCLUSIONS

Our study suggests promise from telemedicine programs provided through employers, specifically when the employer is a local health system and that local health system provides the service to its own employees. If these results could be obtained when a local health system provides the service with similar financial arrangements to non–health system, employer-sponsored plans, the cost of health care in the United States could be reduced.

Author Affiliations: Department of Medicine, University of Pennsylvania (KHC, SJM, DAA), Philadelphia, PA; Leonard Davis Institute of Health Economics, University of Pennsylvania (KHC, SJM, DAA), Philadelphia, PA; Penn Center for Connected Care, University of Pennsylvania Health System (KHC, AMH, SM), Philadelphia, PA; Penn Medicine Center for Health Care Innovation, University of Pennsylvania (KHC, CKS, SJM, DAA), Philadelphia, PA; Department of Biostatistics, Epidemiology, and Informatics, University of Pennsylvania (NM), Philadelphia, PA; University of Pennsylvania Health System (RB, GK), Philadelphia, PA; Office of the Chief Medical Information Officer, University of Pennsylvania Health System (CWH), Philadelphia, PA.

Source of Funding: None.

Author Disclosures: Dr Chaiyachati was employed by Penn Medicine, which offers the studied program as a service, during the completion of this manuscript; is currently employed by and has been a consultant for Verily; is a board member of Intend Health Strategies; has received grants from the National Institute on Aging, the Patient-Centered Outcomes Research Institute, Independence Blue Cross, and Roundtrip, Inc; and has received lecture fees from Villanova School of Business for speaking on a panel on digital health. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (KHC, CKS, SM, GK, SJM, DAA); acquisition of data (KHC, CKS, AMH, SM, RB, CWH); analysis and interpretation of data (KHC, CKS, NM, SM, RB, SJM, DAA); drafting of the manuscript (KHC, CWH, DAA); critical revision of the manuscript for important intellectual content (KHC, CKS, NM, GK, SJM, DAA); statistical analysis (KHC, CKS, NM); provision of patients or study materials (AMH); administrative, technical, or logistic support (KHC, AMH, RB, CWH); and supervision (KHC, CWH, GK).

Address Correspondence to: Krisda H. Chaiyachati, MD, MPH, MSHP, University of Pennsylvania, 423 Guardian Dr, 13th Floor, Blockley Hall, Philadelphia, PA 19104. Email: krisda.chaiyachati@pennmedicine.upenn.edu.

REFERENCES

1. Claxton G, Rae M, Long M, Damico A, Whitmore H. Health benefits in 2018: modest growth in premiums, higher worker contributions at firms with more low-wage workers. Health Aff (Millwood). 2018;37(11):1892-1900. doi:10.1377/hlthaff.2018.1001

2. Claxton G, Rae M, Damico A, Young G, McDermott D, Whitmore H. Health benefits in 2019: premiums inch higher, employers respond to federal policy. Health Aff (Millwood). 2019;38(10):1752-1761. doi:10.1377/hlthaff.2019.01026

3. Claxton G, Damico A, Rae M, Young G, McDermott D, Whitmore H. Health benefits in 2020: premiums in employer-sponsored plans grow 4 percent; employers consider responses to pandemic. Health Aff (Millwood). 2020;39(11):2018-2028. doi:10.1377/hlthaff.2020.01569

4. Claxton G, Rae M, Damico A, Young G, Kurani N, Whitmore H. Health benefits in 2021: employer programs evolving in response to the COVID-19 pandemic. Health Aff (Millwood). 2021;40(12):1961-1971. doi:10.1377/hlthaff.2021.01503

5. Ashwood JS, Mehrotra A, Cowling D, Uscher-Pines L. Direct-to-consumer telehealth may increase access to care but does not decrease spending. Health Aff (Millwood). 2017;36(3):485-491. doi:10.1377/hlthaff.2016.1130

6. Gordon AS, Adamson WC, DeVries AR. Virtual visits for acute, nonurgent care: a claims analysis of episode-level utilization. J Med Internet Res. 2017;19(2):e35. doi:10.2196/jmir.6783

7. Schoenfeld AJ, Davies JM, Marafino BJ, et al. Variation in quality of urgent health care provided during commercial virtual visits. JAMA Intern Med. 2016;176(5):635-642. doi:10.1001/jamainternmed.2015.8248

8. Uscher-Pines L, Mulcahy A, Cowling D, Hunter G, Burns R, Mehrotra A. Access and quality of care in direct-to-consumer telemedicine. Telemed J E Health. 2016;22(4):282-287. doi:10.1089/tmj.2015.0079

9. Hamdy RF, Park D, Dean K, et al. Geographic variability of antibiotic prescribing for acute respiratory tract infections within a direct-to-consumer telemedicine practice. Infect Control Hosp Epidemiol. 2022;43(5):651-653. doi:10.1017/ice.2021.84

10. Martinez KA, Rood M, Jhangiani N, Kou L, Boissy A, Rothberg MB. Association between antibiotic prescribing for respiratory tract infections and patient satisfaction in direct-to-consumer telemedicine. JAMA Intern Med. 2018;178(11):1558-1560. doi:10.1001/jamainternmed.2018.4318

11. Asch DA. The hidden economics of telemedicine. Ann Intern Med. 2015;163(10):801-802. doi:10.7326/M15-1416

12. In April 2021, telehealth utilization falls nationally for third straight month. News release. Fair Health; July 7, 2021. Accessed July 30, 2021. https://www.fairhealth.org/press-release/in-april-2021-telehealth-utilization-falls-nationally-for-third-straight-month

13. Bestanny O, Gilbert G, Harris A, Rost J. Telehealth: a quarter-trillion-dollar post-COVID-19 reality? McKinsey & Company. July 9, 2021. Accessed July 30, 2021. https://www.mckinsey.com/industries/healthcare-systems-and-services/our-insights/telehealth-a-quarter-trillion-dollar-post-covid-19-reality

14. Weiner JP, Bandeian S, Hatef E, Lans D, Liu A, Lemke KW. In-person and telehealth ambulatory contacts and costs in a large US insured cohort before and during the Covid-19 pandemic. JAMA Netw Open. 2021;4(3):e212618. doi:10.1001/jamanetworkopen.2021.2618

15. Barnett ML, Ray KN, Souza J, Mehrotra A. Trends in telemedicine use in a large commercially insured population, 2005-2017. JAMA. 2018;320(20):2147-2149. doi:10.1001/jama.2018.12354

16. Shachar C, Engel J, Elwyn G. Implications for telehealth in a postpandemic future: regulatory and privacy issues. JAMA. 2020;323(23):2375-2376. doi:10.1001/jama.2020.7943

17. The Future of Telehealth: How COVID-19 Is Changing the Delivery of Virtual Care, Hearing Before the Subcommittee on Health of the Committee on Energy and Commerce, 117th Cong, 1st Sess (2021). Accessed August 13, 2021. https://www.congress.gov/event/117th-congress/house-event/LC67702/text?s=1&r=86

18. Mehrotra A, Bhatia RS, Snoswell CL. Paying for telemedicine after the pandemic. JAMA. 2021;325(5):431-432. doi:10.1001/jama.2020.25706

19. Chaiyachati KH, Mahraj K, Mrad CG, et al. Using design and innovation principles to reduce avoidable emergency department visits among employees of a large academic medical center. Healthc (Amst). 2021;9(1):100514. doi:10.1016/j.hjdsi.2020.100514

20. Venkatesh AK, Janke AT, Shu-Xia L, et al. Emergency department utilization for emergency conditions during COVID-19. Ann Emerg Med. 2021;78(1):84-91. doi:10.1016/j.annemergmed.2021.01.011

21. Lei L, Maust DT. Delayed care related to COVID-19 in a nationally representative sample of older Americans. J Gen Intern Med. 2022;37(5):1337-1340. doi:10.1007/s11606-022-07417-4

22. Czeisler M, Marynak K, Clarke KEN, et al. Delay or avoidance of medical care because of COVID-19–related concerns — United States, June 2020. MMWR Morb Mortal Wkly Rep. 2020;69(36):1250-1257. doi:10.15585/mmwr.mm6936a4

23. Clinical Classifications Software Refined (CCSR). Healthcare Cost and Utilization Project. Accessed August 13, 2021. https://www.hcup-us.ahrq.gov/toolssoftware/ccsr/ccs_refined.jsp

24. Rosenbaum PR, Rubin DB. The central role of the propensity score in observational studies for causal effects. Biometrika. 1983;70(1):41-55. doi:10.1093/biomet/70.1.41

25. Thomas L, Li F, Pencina M. Using propensity score methods to create target populations in observational clinical research. JAMA. 2020;323(5):466-467. doi:10.1001/jama.2019.21558

26. Charlson ME, Pompei P, Ales KL, MacKenzie CR. A new method of classifying prognostic comorbidity in longitudinal studies: development and validation. J Chronic Dis. 1987;40(5):373-383. doi:10.1016/0021-9681(87)90171-8

27. Hu Z, Hao S, Jin B, et al. Online prediction of health care utilization in the next six months based on electronic health record information: a cohort and validation study. J Med Internet Res. 2015;17(9):e219. doi:10.2196/jmir.4976

28. Monheit AC. Persistence in health expenditures in the short run: prevalence and consequences. Med Care. 2003;41(suppl 7):III53-III64. doi:10.1097/01.MLR.0000076046.46152.EF

29. Mehrotra A, Prewitt E. Convenient care: opportunity, threat, or both. NEJM Catalyst. July 11, 2019. Accessed July 12, 2019. https://catalyst.nejm.org/convenient-care-opportunity-threat/

30. Lovell T, Albritton J, Dalto J, Ledward C, Daines W. Virtual vs traditional care settings for low-acuity urgent conditions: an economic analysis of cost and utilization using claims data. J Telemed Telecare. 2021;27(1):59-65. doi:10.1177/1357633X19861232