Patient Ratings of Veterans Affairs and Affiliated Hospitals

ABSTRACT

Objectives: Hospital Compare, a website maintained by CMS, allows comparisons of outcomes and processes of care but not of patient satisfaction for hospitals within the Veteran Affairs (VA) Healthcare System. Therefore, we sought to compare online hospital ratings between VA hospitals and their local affiliated hospitals.

Study Design: Observational analysis.

Methods: We identified 39 VA hospitals and a non-VA affiliated hospital with at least 2 online Yelp ratings. We determined the difference in the mean rating (VA-affiliate rating) with weighting by the number of ratings for each hospital. We used multivariate regression to compare mean Yelp ratings between VA and non-VA affiliate hospitals, adjusting for hospital characteristics (bed size, teaching status, and accreditation).

Results: The mean patient rating for VA hospitals (± standard deviation) was higher (3.64 ± 1.0) than the rating for affiliated hospitals (3.09 ± 0.8; P = .0036). There was no significant correlation in rating between a VA hospital and its affiliate (r = 0.07; P = .59). After adjustment for hospital characteristics, the adjusted rating difference (VA-affiliate hospitals) was 0.65 (95% confidence interval, 0.18-1.12).

Conclusions: VA hospitals had higher patient ratings than their non-VA affiliated hospitals, a finding not explained by bed size or teaching status.

Am J Manag Care. 2017;23(6):382-384Takeaway Points

• Patient satisfaction is an important measure of clinical care but cannot be compared between Veterans Affairs (VA) and non-VA hospitals using government data (eg, Hospital Compare).
• Online patient ratings of hospitals (eg, Yelp) have been shown to correlate strongly with more conventional measures of patient satisfaction, as well as patient outcome.
• VA hospitals had significantly higher Yelp ratings than non—VA-affiliated hospitals suggesting better patient satisfaction at VA hospitals.
• The results were not explained by differences in hospital characteristics, such as size and teaching status.

Recently, the Department of Veterans Affairs (VA) started the Choice Program (section 101 of the Veterans Access, Choice, and Accountability Act of 2014), which pays for veterans to receive care at non-VA hospitals if care cannot be provided in a timely manner (within 30 days) at a VA hospital or if the patient lives a significant distance (greater than 40 miles) from one.¹ Hospital Compare, a website maintained by CMS, allows comparisons of outcome and processes of care for VA and non-VA hospitals.² Patient satisfaction is reported on Hospital Compare for non-VA hospitals only. Currently, the only way for patients to compare VA and non-VA hospitals on patient experience is through private online customer rating services (eg, Yelp).

A recent study found a high correlation between Yelp ratings and the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) scores as reported by Medicare on Hospital Compare for non-VA US hospitals.³ Another comparison of HCAHPS and Yelp ratings for non-VA US hospitals found that Yelp ratings covered the main domains of HCAHPS while providing information on additional domains of patient experience.⁴ These data suggest that Yelp may be a reasonable alternative for estimates of the patient experience at VA hospitals.

A common alternative to a VA hospital is an affiliated university hospital. Over 90% of VA hospitals are affiliated with medical schools and many share trainees and physician staff with university hospitals. We sought to compare online hospital ratings between VA and local affiliated hospitals. This allowed us to compare the patient experience at VA and non-VA hospitals in the same region that often share trainees and physicians.

METHODS

Hospital Identification

We searched Yelp for hospital ratings for the 131 VA hospitals from March 2014 to June 2014 and recorded the Yelp ratings 1 (worst) to 5 (best). Ratings were excluded if the category was specific to a department (eg, cafeteria). We only included hospitals if there were 2 or more online ratings. Therefore, we attempted to identify the local university hospital affiliate of each of the VA hospitals. If there was more than 1 local affiliate, we used the affiliate with the greatest number of online reviews. For 1 VA hospital for which there was no local affiliate, we selected a random hospital within 5 miles that had online reviews. We determined hospital characteristics (bed size, membership in the Council on Teaching Hospitals [COTH], presence of an accredited graduate medical education program, and certification by The Joint Commission [TJC]) using data from the American Hospital Association.

Primary Measure of Patient Satisfaction

We determined the mean Yelp rating for each hospital.

Statistical Analysis

Continuous variables of hospital characteristics were compared between VA and non-VA hospitals using t tests. Categorical variables of hospital characteristics were compared using χ² tests. We first compared the mean Yelp ratings of the VA and affiliated hospitals using an unweighted paired t test. In a sensitivity analysis, comparisons of mean ratings were weighted by the number of ratings using analysis of variance. The Shapiro-Wilk test did not reject the assumption of normality for the rating distribution (P = .94). We then examined the correlation (clustering) of online ratings for the pairs of VA and non-VA hospitals. Finding negligible clustering (see results for correlation), we used ordinary least squares multivariate regression to compare mean Yelp ratings, adjusting for hospital characteristics (bed size, membership in COTH, presence of an accredited graduate medical education program, and certification by TJC). We repeated the analyses limiting the sample to those with 5 or more, 10 or more, and 15 or more individual ratings. A 2-sided P <.05 was considered statistically significant. All analyses were performed with Stata version 11.0 (Stata Corp; College Station, Texas).

RESULTS

Hospital Characteristics

We identified 39 VA hospitals (30%) and 39 non—VA-affiliated hospitals with at least 2 Yelp ratings. Hospital characteristics are displayed in the Table. Non-VA hospitals were larger than VA hospitals. The vast majority of both VA and non-VA hospitals were affiliated with medical schools, although non-VA hospitals were more likely to be a member of COTH. Almost all used TJC for hospital accreditation. VA hospitals with less than 2 Yelp reviews (excluded from the study, n = 92) were smaller and less likely to be academic than those VA hospitals with 2 or more ratings (eAppendix Table [eAppendices available at ajmc.com]).

Yelp Ratings

The mean patient rating (± standard deviation) for VA hospitals was higher (3.64 ± 1.0) than the rating for affiliated hospitals (3.09 ± 0.75; P = .0036, t test-unweighted). The median difference between VA and affiliate ratings (VA — non-VA) was 0.5 (interquartile range, –0.5 to 1.5). After weighting for the number of reviews (analysis of variance), the mean patient ratings were slightly higher for both VA hospitals (3.70 ± 0.74) and affiliated hospitals (3.19 ± 0.54), although the difference remained similar and significant (P = .0025). This difference in online ratings persisted when the sample was limited to dyads of hospitals (VA and non-VA) with more than 2 reviews (Figure). There was no significant correlation in rating between a VA hospital and its affiliate (r = 0.07; P = .59). After adjustment for hospital characteristics, the adjusted rating difference (VA-affiliate hospital) was 0.65 (95% confidence interval, 0.18-1.12).

DISCUSSION

Patient satisfaction with care is increasingly recognized as important by payers, including CMS. Hospital ratings on the HCAPHS surveys are publically reported² and used as part of pay-for-performance programs. Patient-initiated reviews have limitations, but a recent study found a strong correlation (r = 0.49; P <.0001) between Yelp reviews and HCAHPS scores for non-VA US hospitals.³ At the time of the study (2011), 25% of all non-VA US hospitals had a Yelp review, and this percentage increased over time—this percentage is consistent with our ability to find reviews for 30% of VA hospitals. The mean score for all non-VA US hospitals was 3.3 in 2011, slightly higher than the non-VA hospital score in our study.

Although Yelp reviews describe patient experience, they may also be a marker of patient outcome. Our study was not powered to detect differences in outcome; however, the Bardach study³ found that high Yelp ratings were significantly associated with lower 30-day all-cause mortality following an admission for myocardial infarction or pneumonia and lower 30-day all-cause readmissions following a discharge for myocardial infarction, heart failure, or pneumonia.

Limitations

Our study is potentially limited by our comparison group, which included local university-affiliated hospitals that often share trainees and faculty with the VA hospital. Although veterans may often choose the local affiliate hospital to receive care, our results may not be applicable to VA comparisons with non-affiliated community hospitals. We excluded 70% of VA hospitals with fewer than 2 reviews and these were smaller and less academic than the included studies sites. A prior study found that smaller hospitals had higher patient satisfaction ratings than larger hospitals although there was no significant impact of teaching status on ratings.⁵ Differences in patients could explain the results. For example, we cannot know if veterans are more likely than nonveterans to rate all hospitals highly.

CONCLUSIONS

We found that VA hospitals have higher patient ratings than their affiliated hospitals, a finding not explained by bed size or teaching status.Author Affiliations: Veterans Affairs Palo Alto Healthcare System (PAH, AZ, NO, AS), Palo Alto, CA; Department of Medicine, Stanford University School of Medicine (PAH, LS)

Source of Funding: US Department of Veterans Affairs HSR&D CHF 04-236.

Author Disclosures: The hospital where Drs Heidenreich, Zapata, Oliva, and Sahay were employed was one of the study sites. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (PAH, AS); acquisition of data (PAH, AZ); analysis and interpretation of data (PAH, AZ, LS, NO, AS); drafting of the manuscript (PAH, LS); critical revision of the manuscript for important intellectual content (PAH, LS, NO, AS); obtaining funding (PAH); administrative, technical, or logistic support (AZ, NO).

Address Correspondence to: Paul Heidenreich, MD, MS, VA Palo Alto HCS, 111C, 3801 Miranda Ave, Palo Alto, CA 94304. E-mail: paul.heidenreich@va.gov.REFERENCES

1. Department of Veterans Affairs. Expanded access to non-VA care through the Veterans Choice Program [interim final rule]. Fed Regist. 2014;79(214):65571-65587.

2. Centers for Medicare and Medicaid Services. Hospital Compare website. http://www.medicare.gov/hospitalcompare. Published 2014. Accessed January 23, 2015.

3. Bardach NS, Asteria-Peñaloza R, Boscardin WJ, Dudley RA. The relationship between commercial website ratings and traditional hospital performance measures in the USA. BMJ Qual Saf. 2013;22(3):194-202. doi: 10.1136/bmjqs-2012-001360.

4. Ranard BL, Werner RM, Antanavicius T, et al. Yelp reviews of hospital care can supplement and inform traditional surveys of the patient experience of care. Health Aff (Millwood). 2016;35(4):697-705. doi: 10.1377/hlthaff.2015.1030.

5. Elliott MN, Cohea CW, Lehrman WG, et al. Accelerating improvement and narrowing gaps: trends in patients’ experiences with hospital care reflected in HCAHPS public reporting. Health Serv Res. 2015;50(6):1850-1867. doi: 10.1111/1475-6773.12305.&ensp;

Download PDF: Patient Ratings of Veterans Affairs and Affiliated Hospitals