CMS Practice Assessment Tool Validity for Alternative Payment Models

ABSTRACT

Objectives: To study the predictive validity of the CMS Practice Assessment Tool (PAT) among 632 primary care practices.

Study Design: Retrospective observational study.

Methods: The study included primary care physician practices recruited by the Great Lakes Practice Transformation Network (GLPTN), 1 of 29 CMS-awarded networks, and used data from 2015 to 2019. At enrollment, trained quality improvement advisers scored each of the PAT’s 27 milestones by its degree of implementation based on interviews with staff, review of documents, direct observation of practice activity, and professional judgment. The GLPTN also tracked each practice’s status regarding alternative payment model (APM) enrollment. Exploratory factor analysis (EFA) was used to identify summary scores; mixed-effects logistic regression was used to assess the relationship between derived scores with APM participation.

Results: EFA revealed that the PAT’s 27 milestones could be summed into 1 overall score and 5 secondary scores. By the end of the 4-year project, 38% of practices were enrolled in an APM. A baseline overall score and 3 secondary scores were associated with increased odds of joining an APM (overall score: odds ratio [OR], 1.06; 95% CI, 0.99-1.12; P = .061; data-driven care quality score: OR, 1.11; 95% CI, 1.00-1.22; P = .040; efficient care delivery score: OR, 1.08; 95% CI, 1.03-1.13; P = .003; collaborative engagement score: OR, 0.88; 95% CI, 0.80-0.96; P = .005).

Conclusions: These results demonstrate that the PAT has adequate predictive validity for APM participation.

Am J Manag Care. 2023;29(2):e58-e63. https://doi.org/10.37765/ajmc.2023.89322

Takeaway Points

• A major goal of the CMS Transforming Clinical Practice Initiative was to help practices prepare to successfully participate in value-based payment mechanisms, such as alternative payment models (APMs).
• Exploratory factor analyses revealed that the CMS Practice Assessment Tool’s 27 milestones could be summed into 1 overall score and 5 secondary scores.
• This overall score and 3 of the secondary scores were associated with increased odds of joining an APM among 632 primary care practices.
• Using these scores (instead of tracking all 27 milestones) would allow for the use of summation dashboards to simplify practice monitoring and benchmarking.

CMS has implemented several large-scale efforts in recent years to promote better care, smarter spending, and better health.¹ One such effort was the Transforming Clinical Practice Initiative (TCPI).² The primary goal of the TCPI was to support clinician practices through nationwide, collaborative, and peer-based learning networks designed to help clinicians and practices to achieve large-scale health care transformation and prepare practices to successfully participate in value-based payment arrangements, such as alternative payment models (APMs).² To assess practices and monitor progress toward value-based APMs, CMS created the Practice Assessment Tool (PAT), which scores practices based on their performance through 27 specific milestones or items.

In response to the TCPI, the Great Lakes Practice Transformation Network (GLPTN) was formed and awarded as 1 of the 29 Practice Transformation Networks (PTNs). GLPTN operated from 2015 through 2019 as a partnership among regional extension centers, Altarum Institute, local health information exchanges, and 8 universities (Indiana University, Purdue University, Northwestern University, University of Kentucky, University of Illinois at Chicago, Rush University, Northern Illinois University, and University of Chicago) to engage practices from Illinois, Indiana, Kentucky, Michigan, and Ohio. This article describes our attempt to identify major and secondary factors that explain PAT score intercorrelations and the predictive validity of PAT score for APM enrollment among 632 primary care practices of the GLPTN.

METHODS

Primary Care Practice Enrollment

During 2015 and 2016, the GLPTN recruited primary care practices for participation, excluding those belonging to an accountable care organization or another PTN and those that did not use electronic health records. Enrollment of an eligible practice into GLPTN was defined by a completed letter of participation and receipt of demographic information for all practice clinicians. For each practice, the GLPTN captured simple description such as the setting of the practice (rural vs urban), its specialty type (family medicine, geriatric medicine, internal medicine, pediatrics, or others), and its size (solo clinician, 2-5 clinicians, 6-15 clinicians, 16-49 clinicians, or ≥ 50 clinicians).

PAT

The PAT has 27 milestones. At the time of enrollment into the GLPTN, trained quality improvement advisers scored each milestone by its degree of implementation based on interviews with practice staff, review of applicable documents and reports, direct observation of practice activity, and professional judgment. Each milestone for primary care practices was scored on a scale of 0 to 3, with 3 indicating the highest performance: not yet implementing that milestone (score 0), getting started with implementation (score 1), implementing and partially operating (score 2), or functioning and performing well (score 3). To promote consistency in assessments, CMS developed materials that were distributed to all advisers. These materials contained detailed instructions for how to assign a score within each milestone, describing specific tasks and results that practices must demonstrate to receive a certain score. Additionally, regional managers had the opportunity to connect with a central coordinating team to discuss scoring and receive additional instruction or clarification if needed.

Enrollment in an APM

Over the course of 4 years, the GLPTN tracked each primary care practice status regarding enrollment in an APM. In the states included in the GLPTN, a variety of APMs were available, either from CMS (Medicare Shared Savings Program, Bundled Payments for Care Improvement, Comprehensive Primary Care Plus) or from private payers (Blue Cross Blue Shield Value Based Partnership and Physician Group Incentive Program, Blue Care Network HMO, and others from Aetna, Anthem, and Humana, to name a few).

Statistical Analysis

Large health care systems had multiple practice assessments per taxpayer identification number (TIN). For all analyses, baseline PAT assessments were reduced to distinct response patterns per practice. Exploratory factor analyses (EFAs) were performed using the SAS FACTOR procedure with the “full” polychoric correlation matrix (ie, values of 1 on the diagonal), principal component estimation method, PROMAX rotation. Scree plots were used to determine the number of factors. Loadings (ie, correlations between milestones and factors) and milestone content, respectively, were examined to determine the statistical strength and conceptual relevance of various factor solutions (ie, loadings for models with different numbers of factors).

Milestone subscale scores were calculated using an unweighted sum of observed scores (range, 0-3) for milestones that clustered together in EFA. All scores for milestone subscales and milestone totals were reported as a percentage (ie, observed score divided by possible score, multiplied by 100), ranging from 0 to 100, with higher scores reflecting better performance.

General linear mixed models were used to compare geographic, practice size, and practice specialty type on PAT scores, including milestone overall score and subscale scores. Each model included a random effect for practice TIN to account for the correlation of assessments within a practice TIN. Two-sided tests were performed at the α of 0.05.

Descriptive statistics were reported for practice characteristics and/or assessment scores by APM participation. Continuous variables were summarized using means and SDs, and categorical variables were presented using frequencies (n) and percentages (%). Chi-square tests were used to test whether APM participation differed by any of the practice characteristics, and a mixed-effects model was used to test whether the derived scales differed by APM participation. Mixed-effects logistic regression models were used to assess the relationship between the derived scales with APM participation while adjusting for other practice characteristics. Each model contained fixed effects for practice characteristics and scale scores, as well as a random effect for practice TIN. All analyses were performed using SAS version 9.4 (SAS Institute Inc).

RESULTS

Among the 632 primary care practices included in the GLPTN, the most common specialty type was family practice (44% of practices), followed by pediatrics (26%), internal medicine (24%), and geriatric (2%); another 5% were classified as other. Approximately 1 in 5 (19%) were in rural areas and half (50%) had 5 or fewer clinicians.

Individual milestones exhibited variation indicated by adequate response counts over the range of their 0-3 scores (Table 1). Milestone score means ranged from 0.4 to 2.4, with most means between 0.5 and 2.0, indicating that practices at baseline were getting started on the milestone activity or implementing with the partial operation. Scree plots indicated that 1 major (overall score) and 5 secondary factors could be used to explain the inter-milestone correlations. We labeled the 5 secondary factors based on the milestones they contained. For example, the first of these factors included 6 milestones, some of which were “patient family engagement: has formal approach to obtaining patient/family feedback,” “QI strategy: organized approach to identify and act on improvement opportunities,” and “transparent measurement and monitoring: regularly produces and share reports on performance,” among others. This first factor was labeled “F1: formal approach to quality improvement.” The other 4 secondary factors were labeled “F2: data-driven care quality,” “F3: collaborative engagement,” “F4: optimal business operations,” and “F5: efficient care delivery.” Table 1 indicates which milestones are included in each factor. Except for the “enhanced access” (milestone 17) (0.38 loading), all milestones exhibited a 1-factor loading above 0.40 (and most were above 0.5), a finding that supports reporting an overall milestone score. Item-total correlations were above 0.30 (except for “enhanced access,” whose correlation was 0.28), indicating additional validity for an overall milestone score. Similarly, good loadings and item-total correlations were revealed for the 5-factor model, which lends predictive validity evidence for the newly created milestone subscale scores.

A total of 240 practices (38%) joined an APM by the end of the 4-year program. The baseline PAT scores revealed that practices that eventually joined an APM scored significantly higher, on average, on the overall score and in 4 of the 5 secondary factor scores than did practices that did not go on to join an APM (Table 2). Analysis of the baseline PAT scores also revealed significant variation in mean scores by practice setting, specialty type, and size. Among specialty type, geriatric practices were the most likely to join an APM (70%); pediatric practices were the least likely (17%) (Table 3). APM participation also varied by practice size, although it was not a linear relationship. Among large practices (≥ 50 clinicians), 64% joined an APM, whereas between 28% and 41% of other practices joined an APM (Table 3).

Logistic regression revealed that the unadjusted total PAT score was associated with significantly higher odds of joining an APM (odds ratio [OR], 1.08; 95% CI, 1.02-1.14; P = .007); when adjusted for other factors, the overall PAT score produced an estimated OR of 1.06, although it was not significant (95% CI, 0.99-1.12; P = .061) (Table 4). When using the 5 secondary factor scores and adjusting for practice characteristics, logistic regression identified that 2 factors, F2: data-driven care quality (OR, 1.11; 95% CI, 1.00-1.22; P = .040) and F5: efficient care delivery (OR, 1.08; 95% CI, 1.03-1.13; P = .003) (Table 5), were associated with significantly higher odds of joining an APM. F3: collaborative engagement was associated with a reduced likelihood of APM participation (OR, 0.88; 95% CI, 0.80-0.96; P = .005), but there may be some collinearity associated with this factor, as it was not significant without the other factor scores in the model. Practice setting (urban vs rural) was not significant in either model, but both specialty type and size were significant in both logistic models.

DISCUSSION

The EFA revealed high factor loadings for 1 major and 5 secondary factors, and baseline PAT factor scores were predictive of transition to an APM. The high factor loadings and item-total correlations support the validity of summarizing the PAT with an overall score and 5 factor scores. Using these factors (instead of tracking all 27 milestones) would allow for the creation of summation dashboards that simplify ongoing monitoring, trending, and benchmarking the practices’ transformation. For practices that wish to monitor and track progress toward APM using the PAT, our results suggest that such programs and/or practices could employ a dashboard of just the factors identified in our analysis instead of needing to continually monitor and evaluate all 27 individual milestones. Dashboards can provide feedback surveillance in real time, helping to mitigate drift off target goals and showcase success. Information from a PAT-based dashboard could signal to programs and/or practices where improvement is needed and which competencies or capabilities require further development. For example, another PTN, the Garden Practice Transformation Network, created a dashboard to summarize and present PTN data. Those results showed that the dashboard engaged practices in actionable discussions about sustainable practice transformation and costs of care.³ Having a tool that uses the overall score and scores from the 5 factors allows for internal assessments and external benchmarking.

We observed univariate and adjusted associations between higher baseline PAT scores and APM participation. Although 1 interpretation is that PAT performance is predictive of APM participation, it is possible that practices with higher PAT scores were those already better suited for transition to an APM model regardless of the influence of the TCPI. In this context, PAT performance and/or a dashboard could serve as a quick way to identify practices not well suited for APM that may have the highest need for technical assistance.

Of the 5 factors identified, 3 were predictive of APM enrollment, including 2 associated with an increased likelihood (data-driven care quality and efficient care delivery) and 1 with a decreased likelihood (collaborative engagement). Although there may be multiple explanations for these associations (or lack thereof), it is notable that the data-driven care quality and efficient care delivery factors involve milestones often associated with capitated payment models or APMs. These include items related to identifying patient risk levels, use of registries, and reductions in unnecessary utilization. Perhaps high scores on these factors reflect practices with infrastructure and processes in place that align with what is needed to thrive in an APM. Conversely, the collaborative engagement factor includes milestones related to community resources, family involvement, and cultivating joy, which are important in care delivery but may not signal administrative readiness for APM participation. However, as we note in the methods, this factor was not significant without the other factor scores in the model, suggesting that there may be some collinearity with this factor. It is not immediately clear why the other 2 factors were not significantly associated with APM participation, but it is possible that they did not offer additional predictive power beyond what the other 3 factors already contributed.

We did not find an association between urban/rural status and APM participation but did observe that the largest practices were more likely to join an APM. Previous studies have reported that providers in urban areas and those owned by or operating within a larger health system are more likely to participate in APMs.^4-7 Some have suggested that smaller, independent practices may lack the resources necessary to participate in APMs.⁷ It may also be that some primary care providers do not see APMs as offering significant cost savings for their patients or they believe that the APMs will not offer sufficient support for robust primary care.⁸ However, research suggests that capitated payment models can help achieve the Quadruple Aim (better care, lower cost, better patient experience, and higher provider satisfaction) in primary care if the models are appropriately risk adjusted and payments are connected to patient-oriented and primary care–oriented quality measures.⁹ Some also call for primary care payment models to cover behavioral health and social services⁹ given the role that primary care plays in many communities and the need to connect medical and social services for disadvantaged populations.

Limitations

The strengths of this analysis include the large geographic region covered by the GLPTN and the variety of practices included. However, our analysis did not assess changes in PAT performance during TCPI participation, which may have improved prediction of APM participation. Additionally, our analysis is limited to TCPI practices and we do not attempt to generalize results to the overall population of all primary care practices. If volunteer participation in the TCPI signals additional readiness for APM participation, this would introduce selection bias that would preclude such a generalization. Further, we did not attempt to ascertain any differences between practices that did and did not choose to participate in the GLPTN. However, half of participating practices were small (≤ 5 clinicians) and baseline milestone scores suggested that most had little experience with practice transformation. Therefore, the results presented here may be most applicable to those types of practices.

CONCLUSIONS

This analysis supports the use of the overall PAT score and the 5 factor subscales to reflect practice progress toward APM readiness.

Author Affiliations: Indiana University School of Medicine (MAB, AJP, POM), Indianapolis, IN; Indiana University School of Nursing (MLD-A), Indianapolis, IN; Indiana University O’Neill School of Public and Environmental Affairs (KIS), Bloomington, IN; University of Michigan Division of Geriatric and Palliative Medicine, Internal Medicine, and Institute for Healthcare Policy and Innovation (CHC), Ann Arbor, MI; Solid Research Group, LLC (CAS), St Paul, MN.

Source of Funding: This work has been partially supported by an award from CMS under its Transforming Clinical Practice Initiative to the Great Lakes Practice Transformation Network (1L1CMS331444-03-00).

Author Disclosures: Dr Boustani is a board member of Acadia Pharmaceuticals, Inc; Eli Lilly and Company; Biogen, Inc; Eisai, LLC; Genentech, Inc; and Merck & Co, Inc; has received grants from the National Institutes of Health, Agency for Healthcare Research and Quality, and Merck & Co; receives author royalties from the books Agile Implementation and The Agile Network; and owns stock in Blue Agilis, LLC; DigiCare Realized, Inc; Preferred Population Health Management (PPHM), LLC; and MyShyft, LLC (formerly known as RestUp, LLC). Dr Simon has attended meetings of the Congressional Budget Office (nonpaid). The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (MAB, KIS); acquisition of data (MAB, KIS); analysis and interpretation of data (MAB, AJP, MLD-A, KIS, CHC, POM); drafting of the manuscript (MAB, MLD-A, KIS, CHC, CAS, POM); critical revision of the manuscript for important intellectual content (MAB, AJP, MLD-A, KIS, CHC, CAS, POM); statistical analysis (MAB, AJP); provision of patients or study materials (MAB); obtaining funding (MAB); administrative, technical, or logistic support (MAB, CAS); and supervision (MAB).

Address Correspondence to: Malaz A. Boustani, MD, MPH, Indiana University School of Medicine, 1101 W 10th St, Indianapolis, IN 46202. Email: mboustan@iu.edu.

REFERENCES

1. Press M, Rajkumar R, Conway P. CMS: innovating to achieve better care, smarter spending, and healthier people. Generations. 2015;39(2):73-77.

2. Transforming Clinical Practice Initiative. CMS. Updated May 4, 2021. Accessed November 10, 2022. https://innovation.cms.gov/innovation-models/transforming-clinical-practices

3. Khanna N, Gritzer L, Klyushnenkova E, et al. Practice transformation analytics dashboard for clinician engagement. Ann Fam Med. 2019;17(suppl 1):S73-S76. doi:10.1370/afm.2382

4. Singh P, Orzol S, Peikes D, Oh EG, Dale S. Participation in the Comprehensive Primary Care Plus Initiative. Ann Fam Med. 2020;18(4):309-317. doi:10.1370/afm.2544

5. Colla CH, Lewis VA, Tierney E, Muhlestein DB. Hospitals participating in ACOs tend to be large and urban, allowing access to capital and data. Health Aff (Millwood). 2016;35(3):431-439. doi:10.1377/hlthaff.2015.0919

6. Lewis VA, Colla CH, Carluzzo KL, Kler SE, Fisher ES. Accountable care organizations in the United States: market and demographic factors associated with formation. Health Serv Res. 2013;48(6, pt 1):1840-1858. doi:10.1111/1475-6773.12102

7. Ouayogodé MH, Fraze T, Rich EC, Colla CH. Association of organizational factors and physician practices’ participation in alternative payment models. JAMA Netw Open. 2020;3(4):e202019. doi:10.1001/jamanetworkopen.2020.2019

8. Bazemore A, Phillips RL Jr, Glazier R, Tepper J. Advancing primary care through alternative payment models: lessons from the United States & Canada. J Am Board Fam Med. 2018;31(3):322-327. doi:10.3122/jabfm.2018.03.170297

9. Park B, Gold SB, Bazemore A, Liaw W. How evolving United States payment models influence primary care and its impact on the Quadruple Aim. J Am Board Fam Med. 2018;31(4):588-604. doi:10.3122/jabfm.2018.04.170388