Cost Savings Associated With Electronic Specialty Consultations

ABSTRACT

Objectives: Electronic consultations, or e-consults, between primary care providers and specialists have been shown to improve access to specialty care, shorten wait times, and reduce outpatient visits. The objective of this study was to evaluate differences in health care costs between patients who received an electronic specialty consultation and patients who received a face-to-face specialty consultation.

Study Design: Retrospective cohort evaluation of patients who received a specialty consultation in the Veterans Health Administration during 2016.

Methods: Patients who received an e-consult were matched 1:1 to patients who received a face-to-face consultation using propensity scores. Total, outpatient, and inpatient health care costs over 3 and 6 months following the specialty consultation were compared using a generalized linear model with a gamma distribution and log link.

Results: e-Consults accounted for 1.8% (urology) to 9.6% (hematology) of specialty consultations, on average. Across 11 specialties, patients receiving an e-consult had significantly lower health care costs compared with patients receiving a face-to-face consultation, ranging from 3.6% (cardiology) to 30.7% (hematology) lower. This was largely driven by differences in outpatient costs. Patients receiving an e-consult had significantly lower outpatient costs for all specialties except cardiology, ranging from 6.9% (endocrinology) to 31.2% (hematology) lower. Three-month inpatient costs among those who received an e-consult were significantly lower only in cardiology (5.2%), nephrology (9.3%), pulmonary (13.0%), and gastroenterology (14.3%).

Conclusions: Electronic specialty consultations are a potential mechanism to reduce health care costs and promote the efficient use of health care resources.

Am J Manag Care. 2021;27(1):e16-e23. https://doi.org/10.37765/ajmc.2021.88579

Takeaway Points

• Patients receiving electronic rather than face-to-face specialty consultations had lower 3-month health care costs across 11 specialties, ranging from 3.6% lower in cardiology to 30.7% lower in hematology.
• The cost differences between patients receiving electronic vs face-to-face specialty consultations were largely driven by differences in outpatient costs.
• Electronic consultations between primary care providers and specialists may be a promising mechanism to reduce health care costs.

Electronic consultations, or e-consults, are non–face-to-face consultations conducted between clinicians through a shared electronic health record or web-based platform.¹ They are most commonly used by primary care providers (PCPs) to solicit specialty input for their patients. e-Consults have been adopted across several health care systems to increase access to specialty care services, reduce burden on patients, and potentially reduce health care costs. Recent research results have shown that e-consults can reduce outpatient specialist utilization without reducing access to specialty expertise.^2-5 However, it is unknown whether e-consults influence health care spending.

e-Consults could potentially reduce health care costs by facilitating efficient access to specialty care services and reducing unnecessary face-to-face visits. However, e-consults may increase health care costs if they lead to unnecessary diagnostic testing, provide lower-quality care, or delay (not supplant) the need for face-to-face visits. A 2018 publication by Anderson and colleagues examined the association between cardiology e-consults and health care costs for 369 Medicaid recipients cared for at a federally qualified health care center.¹ They reported an approximate $466 cost savings per patient receiving a cardiology e-consult.¹ However, evidence is lacking that examines the association between e-consults and health care costs for other specialties, among larger samples, and for payer types besides Medicaid.

The Veterans Health Administration (VHA) is the largest integrated health care system in the United States and provides primary and specialty care for more than 9 million veterans annually.⁶ An e-consult program was piloted among several VHA medical centers in 2009 and expanded nationally in 2011. More than 200,000 e-consults across 11 specialties were completed in the VHA in 2016. The objective of this retrospective cohort evaluation was to compare health care costs between patients who received a specialty care e-consult and patients who received a face-to-face specialty consult.

METHODS

Population

The population for this evaluation included patients who had a specialty care consultation, either electronically or face to face, in the VHA during calendar year 2016. Specialty care services are typically housed at one of the VHA medical centers, whereas primary care can be obtained from the medical center or community-based outpatient clinics. VHA PCPs request face-to-face or electronic specialty consultations by placing an order through the electronic health record. For e-consults, the reason for the consult, and any accompanying information, is recorded as free text. The specialist reviews this information and responds to the PCP electronically by providing their recommendation or by requesting additional information or testing. In some cases, the specialist may request a face-to-face visit with the patient.

Exposure

A specialty consultation, which was further categorized as an e-consult or a face-to-face consultation, defined the exposure. Health care utilization data from the VHA’s Corporate Data Warehouse were used to identify e-consults and their associated patients and providers. We first identified patients who received an e-consult for 1 of the following 11 specialties: cardiology, diabetes, endocrinology, gastroenterology, hematology, neurology, oncology, pain medicine, pulmonary, nephrology, and urology. These 11 specialties were selected to align with prior e-consultation literature,⁷ are all uniquely identified from stop codes within VHA data, and collectively represent very different clinical care, with some potentially being more ideal for e-consults than others.

Patients in the e-consult cohort were then matched 1:1 to patients who received a face-to-face specialty consult. To identify a similar population to the e-consult cohort, propensity score matching was used to select patients for the face-to-face cohort. Patients were matched on individual characteristics (age, gender, marital status, and race), Medicaid eligibility (as a proxy for income and availability of secondary insurance), specialty of consultation, and severity (as measured by comorbidities, survival, and total health care costs prior to the index date). The date of the e-consult or face-to-face specialty visit served as the index date.

Outcomes

The primary outcomes were total health care costs at 3 and 6 months after the index specialty consultation. Secondary outcomes included inpatient and outpatient costs at 3 and 6 months. Data from the VHA’s Managerial Cost Accounting (MCA) system were used to calculate the health care costs for patients in each cohort.⁸ Cost data in the MCA system are based on the allocation of actual resources to an encounter.⁸ Total health care costs, total outpatient health care costs, and total inpatient health care costs were compared between the e-consult and face-to-face cohorts for 3 and 6 months after the index date. This time frame aligned with that in prior literature¹ and was informed by clinician input on the duration of expected influence from a single e-consult. Patients with outlier 6-month costs were identified using the interquartile rule for outliers⁹ and were removed from the analysis. For patients in the analytic data set, the health care costs for 1 month prior to the index visit and for 6 months after the index visit were calculated at the monthly level.

Analysis

We first quantified e-consult use across specialties by dividing the number of e-consults by the sum of e-consults and face-to-face specialty visits for each clinic. Our economic evaluation took the perspective of the VHA health care system and focused on direct health care costs accrued to the VHA health care system. For the cost outcomes, we calculated medians and 95% CIs for those receiving e-consults and those receiving face-to-face specialty consultations. Differences between the 2 cohorts were tested through Mann-Whitney U tests. This nonparametric test was used due to the positive skew observed with the cost data. Analyses were stratified by specialty of care and thus results were presented separately for each specialty. Total, outpatient, and inpatient health care costs were also evaluated using a generalized linear model with a gamma distribution and log link to account for the skewed cost data.¹⁰ The multivariable regression models controlled for patient characteristics (age, gender, and race), patient severity (as measured by the Charlson Comorbidity Index and dichotomous indicator variables for the individual comorbidities within the index), location (urban/rural location and distance to the nearest VHA facility), and VHA facility characteristics (facility complexity and a wage index¹¹ to account for geographic variation in costs). Standard errors from the models were clustered at the facility level to account for potential correlation within a VHA facility.¹² The dependent variables included the outcomes of interest detailed above. The primary independent variable was specialty visit categorization (ie, electronic or face to face). All models were estimated separately for each of the 11 specialties.

During the period of the study, veterans enrolled in VHA health care were eligible to receive specialty care from non-VHA providers and health systems through the Veterans Choice Program if they resided more than 40 miles away from a VHA provider or had to wait more than 30 days for an appointment.¹³ In an effort to identify a subset of the population who did not receive community care, a sensitivity analysis was conducted that restricted the patients in the analysis to those residing within 40 miles of a VHA facility.

RESULTS

Use of e-Consults

The number of e-consults conducted in calendar year 2016 are provided in Table 1 [part A and part B]; they ranged from 4892 in the diabetes specialty to 42,141 in the cardiology specialty. Across sites, the average uptake of e-consults ranged from 1.8% of total visits in urology to 9.6% of total visits in hematology. Table 1 also reports the characteristics of patients receiving an e-consult vs face-to-face specialty consultation following propensity score matching. After matching, there are still statistically significant but overall small differences between the 2 patient populations. For example, among those who received cardiology specialty visit consultation, the mean age of patients was 67.2 years and 67.8 years in the e-consult and face-to-face cohorts, respectively (P < .001).

Total Health Care Costs

Median 3- and 6-month total health care costs, stratified by specialty, are provided in Table 2. Patients receiving consultation for the cardiology specialty had the highest costs among the e-consult cohort, with median 3- and 6-month health care costs of $3312 and $6156, respectively. Patients receiving consultation for the diabetes specialty had the lowest costs among the e-consult cohort, with median 3- and 6-month health care costs of $2264 and $4150, respectively. In the face-to-face cohort, 3-month health care costs were highest for patients receiving consultation for the diabetes specialty ($3431); 6-month health care costs were highest for patients receiving consultation for the oncology specialty ($7109). Three-month health care costs were lowest for patients receiving face-to-face consultations for the nephrology specialty ($3196); 6-month health care costs were lowest for patients receiving face-to-face consultation for the urology specialty ($5281). After 3 and 6 months, health care costs were significantly higher in the face-to-face cohort compared with the e-consult cohort for all specialties except cardiology. Within the cardiology specialty, there was no significant difference in total costs at 3 months, but costs were significantly lower among patients who received e-consults at 6 months.

Results were similar when using generalized linear models and adjusting for covariates. Table 3 reports the multivariable regression model findings for 3- and 6-month total health care costs. Patients in the e-consult cohort had significantly lower 3-month health care costs across all specialties. Total cost differences ranged from 3.6% lower among patients receiving cardiology e-consults to 30.7% lower among those receiving hematology e-consults. Patients in the e-consult cohort had significantly lower 6-month health care costs across all specialties, except in cardiology, where no significant cost differences were observed between cohorts. Among the remaining 10 specialties, total cost differences ranged from 10.0% lower among patients receiving endocrinology e-consults to 39.7% lower among those receiving hematology e-consults.

Outpatient Costs

The majority of the cost differences were due to differences in outpatient costs rather than inpatient costs. Table 4 presents the 3- and 6-month outpatient cost differences for the e-consult and face-to-face cohorts, stratified by specialty. Patients in the e-consult cohort had significantly lower 3-month outpatient costs for all specialties, except cardiology, ranging from 6.9% lower in endocrinology to 31.2% lower in hematology. At 6 months, costs were significantly lower for patients in the e-consult cohort for all specialties, except cardiology. Hematology once again had the largest comparative difference in outpatient costs.

Inpatient Costs

Table 5 presents the 3- and 6-month inpatient cost differences for the e-consult and face-to-face cohorts, stratified by specialty. There were few significant differences in inpatient costs between the 2 cohorts. In comparison with those of the face-to-face cohort, the 3-month inpatient costs among those who received an e-consult were significantly lower in cardiology (5.2% lower), gastroenterology (14.3% lower), pulmonary (13.0% lower), and nephrology (9.3% lower). At 6 months, inpatient costs were 15.7% lower among patients who received hematology e-consults and higher among those receiving urology (16.7%) and cardiology (20.6%) e-consults.

More than 98% of the patients in the analysis lived within 40 miles of a VHA medical center. Results of the sensitivity analysis including only those patients were similar to the findings presented above, suggesting that the results were robust even without the direct inclusion of community care cost data.

DISCUSSION

The purpose of this evaluation was to examine the influence of e-consults on subsequent health care costs. This evaluation found that patients who received specialty e-consultation rather than face-to-face specialty consultation had significantly lower total health care costs following the consultation. This finding was observed within each of the 11 studied specialties at 3 months. This translates to 3-month cost savings ranging from $120 (cardiology) to $1100 (oncology) for each patient referred for an e-consult rather than a face-to-face consultation. Multiplying the number of e-consults by the difference in total cost for each specialty, this equates to an annual savings of approximately $110 million. This is the first large-scale evaluation that assesses health care costs associated with e-consults across a variety of specialties among a nationwide sample.

Cost savings among the e-consult cohort were largely driven by differences in outpatient rather than inpatient costs. This is likely because outpatient costs accounted for the majority of total costs at 3 and 6 months. Unfortunately, the cost data used to inform this analysis were aggregated to the patient-month and therefore we did not have specific health care utilization for each patient. Future work will evaluate specific health care utilization differences between e-consults and face-to-face specialty visits to more clearly identify what is driving these cost differences. We also observed large variation in e-consult utilization and cost savings between specialties. Differences in utilization across specialties may be attributable to differing levels of support and infrastructure provided to implement e-consults across facilities and specialties. It is also likely that some specialties are simply more aligned to provide care without the need for a face-to-face visit. For example, specialties that are heavily reliant on physical examination of patients, taking detailed and condition-specific clinical histories, or performing clinical procedures may be less likely to rely solely on electronic communication with other providers to make clinical decisions.

When interpreting these findings, it is important to also consider the operational cost of providing the e-consult, which primarily equates to PCP and specialist time. In the VHA, providers receive 1 of 3 levels of workload credit based on the time spent providing the e-consult.⁶ The workload accounting is then figured into how much money each VHA facility is allocated. Outside of the VHA, multiple potential reimbursement mechanisms exist for the operational cost of e-consults. For example, some hospitals provide salary support to providers to compensate their effort on e-consults. At the Mayo Clinic, e-consults are scheduled as 15-minute appointments and the providers then receive visit credit. Insurance companies could reimburse as traditional fee-for-service, and some state Medicaid programs provide a transactional payment for either the PCP, the specialist, or both.⁶ As e-consults are integrated and expanded in the health care system, the operational cost and potential reimbursement must be considered.

The findings from this analysis expand on a publication by Anderson and colleagues¹ that evaluated differences in 6-month total health care costs between 369 Medicaid patients who were randomized to receive either a cardiology e-consult or a face-to-face visit. They found that 6-month health care costs were $466 lower for patients who had an e-consult, with the majority of the cost savings occurring in residual claims.¹ Our evaluation found that 3-month health care costs were approximately $120 lower for veterans who had a cardiology e-consult, with the majority of the cost savings occurring in outpatient care. Although this number seems small, this translates to potentially substantial cost savings when considered within the context of an entire health care system, as more than 40,000 cardiology e-consults occurred within the VHA in 2016. Further, e-consults represented only approximately 4% of cardiology specialty care. With the potential for e-consult use to increase, savings could reach the millions of dollars for the cardiology specialty alone.

This work also adds to prior literature on e-consults and increasing access to specialty care by providing evidence that increasing access to specialty care through e-consults is also likely an efficient use of resources. This has unique implications for the VHA with the implementation of the MISSION Act in June 2019.¹⁴ The intent of the MISSION Act is to provide veterans with more health care options. The current system is structured so that it is easier for PCPs to refer eligible patients to the community, rather than referring patients to a specialist within the VHA. e-Consults could be an option to facilitate timeliness of access and specialty care recommendations within the VHA system. The speed of an e-consult is likely much faster than if the patient were to seek care outside of the VA. In addition to increasing the timeliness of specialty care access, e-consults would also decrease coordination efforts that are required when a veteran obtains care outside of the VHA system, and they would promote patient-centricity by precluding the patient’s need to drive to (or be transported to) additional health care services.

Given the coronavirus disease 2019 (COVID-19) global pandemic, the findings from this study are particularly timely and clinically relevant, as the importance of decreasing in-person face-to-face contact for patients and providers continues to increase. e-Consults are likely one mechanism that can help reduce face-to-face contact between patients and providers to reduce potential transmission of the virus that causes COVID-19. Given the unknown time course of the current COVID-19 pandemic, and that systems are continuing to encourage the adoption of non–face-to-face modalities, this may result in sustained and increased use of e-consults.

Although e-consultation may be a promising mechanism to deliver efficient specialty care, reduce health care costs, and reduce the need for face-to-face contact, a number of questions should be evaluated with further adoption. We have demonstrated significant cost savings at 3 and 6 months, but future work will be important to verify that the cost savings extend to the long term, to ensure that e-consultation is not simply delaying inevitable care. One limitation of our study is that we were unable to address the possibility of indication bias, in that patients might simply be referred for an e-consult when providers have simpler (ie, less costly) clinical questions. Future research should aim to compare the costs between consultation mechanisms among patients referred to specialty consultation for the same reasons (eg, atrial fibrillation medication management). Although prior work has shown that providers have reported favorably on the quality of care delivered to patients through e-consults, formal assessments of quality of care across a range of conditions have yet to be conducted within the VHA or elsewhere. It will be important to assess the impact of delivering e-consults for PCPs, specialists, and other clinical staff. Preliminary qualitative evaluations made by VHA PCPs have indicated that the e-consult process can be burdensome and may increase their clinical workload. Given the high level of burnout among VHA PCPs and the relationship between burnout and workforce turnover, potential unintended consequences of e-consult adoption should be considered. This would be consistent with other research showing that technological interventions, such as e-consults, may contribute to burnout among physicians.¹⁵ Lastly, we know little about patient perceptions regarding e-consults and how this mechanism of care may affect patient satisfaction and their perceptions of care quality. Some published research results suggest that patients, PCPs, and specialty physicians report being satisfied with e-consults due to the improved communication and timeliness of care; however, this work was limited in sample size, with data from 1 health system, and it included providers who had chosen to use e-consults voluntarily.¹⁶ This is a ripe and necessary area for additional research, especially as providers experience an increase in pressure to pursue e-consults and other non–face-to-face modalities.

Limitations

Our evaluation has limitations, primarily related to the potential for residual confounding within our results. It is possible that the patients in the e-consult cohort had less severe (and thus less costly) conditions, and thus the PCP felt comfortable ordering an e-consult rather than a face-to-face specialty consultation. We matched patients in the e-consult cohort to patients in the face-to-face cohort through propensity score matching that adjusted for important clinical and sociodemographic characteristics that could affect cost. For example, we matched on health care costs prior to the index visit to balance the 2 cohorts on health care utilization. If patients in the face-to-face cohort had more severe conditions than patients in the e-consult cohort, we would expect their prior month costs to be higher; however, in 9 of the 11 specialties, the prior month health care costs were significantly equivalent or lower in the face-to-face cohort. We also matched on comorbidities and survival to promote groups that were similar in condition severity. Before matching, 1-year mortality was similar between the 2 groups (9.9% among patients who received a face-to-face consultation and 8.6% among patients who received an e-consult), which thus does not suggest that patients who had more severe conditions were reserved for face-to-face consultations. Further, this evaluation included only costs of health care provided by the VHA. During the evaluation time period, veterans could receive care from a community, non-VHA provider through the Veterans Choice Program if they satisfied select criteria, including living further than 40 miles from a VHA provider.¹³ Our analysis accounted for this by controlling for distance in all adjusted regression models and conducting a sensitivity analysis that included only patients who lived within 40 miles of a VHA facility and who thus would not be eligible for reimbursed community care based on the distance criteria. The results were consistent among our sensitivity analysis and complete analysis, suggesting that the cost savings would likely persist even if data on cost of community care were included. Similarly, the economic perspective of our evaluation was that of the VHA health care system and thus did not include potential costs to the patient or other members of society. There could be other tangible cost savings with e-consults, including patient travel expenses to a face-to-face visit and patient and/or caregiver time missed from work to attend a face-to-face visit. These potential cost savings were not included in this analysis and should be considered additive to the savings reported in this evaluation. Accordingly, the VHA may accrue additional cost savings if its reimbursements to patients for eligible patient travel expenses were reduced due to fewer face-to-face visits.

CONCLUSIONS

In this study of more than 200,000 specialty consultations occurring within the VHA in 2016, patients referred to specialty care through an e-consult rather than face to face had lower 3- and 6-month costs across nearly every specialty studied. Given that prior study results have shown that e-consults also represent a way to improve access to specialty health care, reduce appointment wait times, and reduce face-to-face consultations, this study provides additional, compelling support for the continued investigation into e-consults as a potential mechanism to deliver high-value and lower-cost care. Future work is needed to estimate longer-term costs and then compare the quality of care between e-consult and face-to-face mechanisms.

Acknowledgments

This work was funded by the VA Office of Rural Health and sponsored by the VA Office of Veterans Access to Care, Department of Veterans Affairs, Washington, DC, through a MyVA Access Improvement Project Grant: “VISN19 VA Denver Developing best practices for subspecialty e-consultation procedures.” The activities were undertaken in support of a Veterans Health Administration (VHA) operational project and did not constitute research, in whole or in part, in compliance with VHA Handbook 1058.05. Therefore, institutional review board approval was not required. The views expressed in this article are those of the authors and do not necessarily represent the views of the US Department of Veterans Affairs. The authors have no conflicts of interest related to this content to report.

Author Affiliations: Center of Innovation for Veteran-Centered and Value-Driven Care, VA Eastern Colorado Healthcare System (MDW, PMH, RRK, JS), Aurora, CO; Department of Medicine, University of Colorado Anschutz Medical Campus (PMH, JS), Aurora, CO; School of Medicine, Case Western Reserve University (SRK), Cleveland, OH; Veterans Affairs Central Office (SRK), Washington, DC; Center of Innovation for Veteran-Centered and Value-Driven Care, VA Puget Sound Health Care System (JT-S, DHA), Seattle, WA.

Source of Funding: This work was funded by the Veterans Health Administration Office of Rural Health and sponsored by the Veterans Health Administration Office of Veterans Access to Care through a MyVA Access Improvement Project Grant.

Author Disclosures: The authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (MDW, SRK, JS); acquisition of data (JT-S, DHA); analysis and interpretation of data (MDW, PMH, SRK, JS); drafting of the manuscript (MDW, RRK, JT-S, DHA, JS); critical revision of the manuscript for important intellectual content (MDW, PMH, SRK, JS); statistical analysis (MDW); obtaining funding (RRK, DHA, JS); administrative, technical, or logistic support (PMH, RRK, JT-S); and supervision (DHA, JS).

Address Correspondence to: Melanie D. Whittington, PhD, Center of Innovation for Veteran-Centered and Value-Driven Care, VA Eastern Colorado Healthcare System, 1700 N Wheeling St, Aurora, CO 80045. Email: Melanie.whittington3@gmail.com.

REFERENCES

1. Anderson D, Villagra V, Coman EN, et al. A cost-effectiveness analysis of cardiology eConsults for Medicaid patients. Am J Manag Care. 2018;24(1):e9-e16.

2. Chen AH, Kushel MB, Grumbach K, Yee HF Jr. Practice profile: a safety-net system gains efficiencies through ‘eReferrals’ to specialists. Health Aff (Millwood). 2010;29(5):969-971. doi:10.1377/hlthaff.2010.0027

3. Kim EJ, Orlander JD, Afable M, et al. Cardiology electronic consultation (e-consult) use by primary care providers at VA medical centres in New England. J Telemed Telecare. 2019;25(6):370-377. doi:10.1177/1357633X18774468

4. Kim-Hwang JE, Chen AH, Bell DS, Guzman D, Yee HF Jr, Kushel MB. Evaluating electronic referrals for specialty care at a public hospital. J Gen Intern Med. 2010;25(10):1123-1128. doi:10.1007/s11606-010-1402-1

5. Barnett ML, Yee HF Jr, Mehrotra A, Giboney P. Los Angeles safety-net program eConsult system was rapidly adopted and decreased wait times to see specialists. Health Aff (Millwood). 2017;36(3):492-499. doi:10.1377/hlthaff.2016.1283

6. Vimalananda VG, Gupte G, Seraj SM, et al. Electronic consultations (e-consults) to improve access to specialty care: a systematic review and narrative synthesis. J Telemed Telecare. 2015;21(6):323-330. doi:10.1177/1357633X15582108

7. Kirsh S, Carey E, Aron DC, et al. Impact of a national specialty e-consultation implementation project on access. Am J Manag Care. 2015;21(12):e648-e654.

8. Managerial cost accounting (MCA). US Department of Veterans Affairs. Accessed August 29, 2019. https://www.herc.research.va.gov/include/page.asp?id=managerial-cost-accounting

9. Pirson M, Dramaix M, Leclercq P, Jackson T. Analysis of cost outliers within APR-DRGs in a Belgian general hospital: two complementary approaches. Health Policy. 2006;76(1):13-25. doi:10.1016/j.healthpol.2005.04.008

10. Berkowitz SA, Seligman HK, Rigdon J, Meigs JB, Basu S. Supplemental Nutrition Assistance Program (SNAP) participation and health care expenditures among low-income adults. JAMA Intern Med. 2017;177(11):1642-1649. doi:10.1001/jamainternmed.2017.4841

11. Medicare wage index for VA facilities. US Department of Veterans Affairs. Accessed August 29, 2019. https://www.herc.research.va.gov/include/page.asp?id=guidebook-wage-index

12. Desai M, Bryson SW, Robinson T. On the use of robust estimators for standard errors in the presence of clustering when clustering membership is misspecified. Contemp Clin Trials. 2013;34(2):248-256. doi:10.1016/j.cct.2012.11.006

13. 10 things about the Veterans Choice Program. US Department of Veterans Affairs. July 25, 2017. Accessed August 29, 2019. https://www.va.gov/health/newsfeatures/2017/july/things-to-know-about-the-veteran-choice-program.asp

14. VA launches new health care options under MISSION Act. News release. US Department of Veterans Affairs.June 6, 2019. Accessed June 6, 2019. https://www.blogs.va.gov/VAntage/61286/va-launches-new-health-care-
options-mission-act/

15. Gleason N, Ackerman S, Shipman SA. eConsult—transforming primary care or exacerbating clinician burnout? JAMA Intern Med. 2018;178(6):790-791. doi:10.1001/jamainternmed.2018.0762

16. Rodriguez KL, Burkitt KH, Bayliss NK, et al. Veteran, primary care provider, and specialist satisfaction with electronic consultation. JMIR Med Inform. 2015;3(1):e5. doi:10.2196/medinform.3725