Identifying Complex Patients Using Adjusted Clinical Groups Risk Stratification Tool

ABSTRACT

Objectives: To produce an efficient and practically implementable method, based on primary care data exclusively, to identify patients with complex care needs who have problems in several health domains and are experiencing a mismatch of care. The Johns Hopkins ACG System was explored as a tool for identification, using its Aggregated Diagnosis Group (ADG) categories.

Study Design: Retrospective cross-sectional study using general practitioners’ electronic health records combined with hospital data.

Methods: A prediction model for patients with complex care needs was developed using a primary care population of 105,345 individuals. Dependent variables in the model included age, sex, and the 32 ADGs. The prediction model was externally validated on 30,793 primary care patients. Discrimination and calibrations were assessed by computing C statistics and by visual inspection of the calibration plot, respectively.

Results: Our model was able to discriminate very well between complex and noncomplex patients (C statistic = 0.9; 95% CI, 0.88-0.92), whereas the calibration plot suggests that the model provides overestimates of complex patients.

Conclusions: With this study, the ACG System has proven to be a useful tool in the identification of patients with complex care needs in primary care, opening up possibilities for tailored interventions of care management for this complex group of patients. Utilizing ADGs, the prediction model that we developed had a very good discriminatory ability to identify those complex patients. However, the calibrating ability of the model still needs improvement.

Am J Manag Care. 2022;28(4):e140-e145. https://doi.org/10.37765/ajmc.2022.88867

Takeaway Points

The Johns Hopkins ACG System appears to hold promise for identifying a complex group of patients with mismatched care.

• Aggregated Diagnosis Groups—the ACG System’s categorization of diagnosis types—seem to be an effective multimorbidity marker, which can be used in primary care to identify complex patient groups.
• Identifying complex groups of patients in primary care is the first step in providing appropriate and proactive care management. This is a key component in population health management approaches.
• Linkage of other data sources, especially social data, to general practitioners’ data will provide complete patient profiles, allowing general practitioners to provide more adequate care.

As populations age and the presence of multimorbid and complex patients becomes the norm, the pressure on health systems in terms of workload and costs is immense.¹ Single-disease management approaches are no longer sufficient to meet the needs of an increasing number of complex patient groups who need care oriented toward their overall health.² In addition, strategies distinguishing different levels of complexity within a population are desirable. Population health management (PHM) approaches aim to allocate available health resources to the appropriate patient groups within the population. Risk stratification tools, such as the widely used Johns Hopkins ACG (formerly Adjusted Clinical Groups) System, play an important role in the identification of specific patient groups for PHM, aiming to identify subgroups in whom avoidable adverse health events could be prevented. With predictive modeling, high-risk patients can be successfully selected for extensive and proactive care management programs.³

One group for whom it seems beneficial to set up a PHM approach, including risk stratification, is that of patients with complex care needs, who have medical problems involving multiple health domains and experience a mismatch of care offerings with their needs. Often the consequence of this mismatch is high care utilization—in particular, expensive and undesirable care, such as emergency or unplanned care. This group of patients was first described by Atul Gawande, MD, MPH, in his 2011 article in The New Yorker, “The Hot Spotters.”⁴ For this group of patients, it would seem that a multidisciplinary and personalized approach would be advantageous, but evidence for the effectiveness of this kind of approach is still ambiguous.⁵ One of the reasons is the incorrect assignment of patients to this intense but effective individualistic approach, leading to a greater mismatch in care. Hence, the first step in providing those complex patients with the appropriate necessary care is a practical and efficient identification of the population who is most likely to benefit from the intervention.

Different methods to identify patients with complex care needs have been utilized by different organizations, using various criteria and types of data sources. The heterogeneity of high-cost patients, including patients with unpreventable costs, makes a general identification based on claims data inefficient.^6,7 A focus on hospitalizations or emergency department (ED) use has proven to be more effective.⁸ However, a complete profile of complex patients requires complete health profile data. In most health care systems, patient data are registered in fragmented silos with significant data linkage challenges; a wide application of risk stratification tools, based on complete patient profiles, is therefore limited. Clinically based predictive models using medication and diagnostic data are especially efficient for prospectively identifying candidates for care management programs.⁹ Especially in primary care–led health systems, in which primary care physicians function as gatekeepers, we have the opportunity to look at more complete health profiles of patients, including medication and diagnostic data, without linkage of data between silos. Unfortunately, despite this great opportunity, evidence for validated risk stratification models to identify patients with complex care needs in primary care is lacking.

Various validated models to map different levels of multimorbidity are available. Because multimorbidity plays an important role in complex patients, the level of multimorbidity is a useful tool, in conjunction with hospitalization and ED visits, in identifying these patients. Strong evidence exists for the ACG System as a tool to determine care burden or multimorbidity, as well as risks for hospitalization and ED utilization.^10-12 With this research, we explored the possibilities of the ACG System as a potential tool to identify patients with complex care needs in primary care.

This study’s aim is to produce an identification method for patients with complex care needs using primary care data in order to perform a PHM approach. Our goal was to answer this research question: Is a prediction model using the ACG risk stratification tool to identify patients with complex care needs, defined as patients with problems in multiple health domains and high acute hospital care, statistically valid for use in primary care?

METHODS

Study Design and Study Population

This work was designed as a retrospective cross-sectional study. To identify patients with complex care needs, a prediction model was developed and externally validated using 2 study populations from 2 different regions in the Netherlands. Population 1 (n = 105,345) and population 2 (n = 30,793) were used as training and validation sets, respectively. To ensure completeness of primary care data, patients were included only if they were registered with a participating general practice for the complete period from January 2016 to December 2016. Because dates for registration and deregistration are unreliable within general practitioners’ electronic health records (EHRs), the period of registration was established using reimbursed registration fees. For each registered patient, a registration fee is reimbursed by the general practice each quarter. Because registration fees are recorded very well within the EHRs, patients were included only when 4 registration fees for 2016 were reimbursed, indicating that the person was registered with the general practice for the complete year. In eAppendix A (eAppendices available at ajmc.com), all registration fee codes nationally used in the Netherlands are presented. In addition, patients were included only when linkage with the database of Statistics Netherlands, the Dutch central bureau for statistics, was possible.

For ethical reasons, deceased patients were excluded from this study.

Data and Linkage

For this study, data from different data sources were linked anonymously at the individual patient level. Data extracted from EHRs of participating general practices included individuals’ general information and information on diagnoses and medications from January to December 2016. Encryption of the EHR data was performed by Statistics Netherlands under strict rules to secure individuals’ privacy. A record identification number (RIN) was assigned to each individual based on birth date, gender, and complete postal code. With the RINs, linkage of EHR data with Statistics Netherlands’ microdata was made possible within the secured environment of Statistics Netherlands. Microdata comprise different types of non–publicly available data on an individual level. Under strict conditions, these microdata are accessible for statistical and scientific research. For this study, we linked the encrypted EHR data to the number of acute care visits extracted from the Dutch medical specialty information system, available as microdata within the secured environment of Statistics Netherlands. Acute care visits were defined as acute hospital care visits to the ED or to another hospital department in which an emergency care practitioner was involved.

Definition of Patients With Complex Care Needs

Adapted from Gawande’s “The Hot Spotters” definition, 2 prerequisites were set to define patients with complex care needs. The first prerequisite concerns having health problems in at least 2 of 3 different health domains registered within primary care. Problems relating to the chronic physical, the mental, and the social domains were identified by selecting corresponding International Classification of Primary Care version 1 (ICPC-1) diagnosis codes within the EHR data. eAppendix A gives an overview of the ICPC-1 codes for all 3 domains. The second prerequisite concerns having at least 2 acute care visits in a 12-month period (January-December 2016), considered high acute care utilization. Acute care visits were identified from health care activity codes within the medical specialty data set available as microdata. Health care activity codes are used nationwide by health care insurers and providers. In eAppendix A, the specific codes corresponding with acute health care use are presented.

Statistical Analysis

To assess the ACG System as an identification method for patients with complex care needs, we developed a prediction model by using logistic regression analysis in the first study population and then externally validated the model in the second study population. To investigate the similarity of both study populations, individuals’ characteristics were compared by using χ² and independent t tests for categorical and continuous variables, respectively.

Dependent variables. The dependent variable, or the outcome, in the logistic regression analysis was whether a person was identified as a complex patient (as described above).

Independent variables. Aggregated Diagnosis Groups (ADGs), the ACG System’s categorization of diagnosis types, were used as independent variables. ICPC-1 diagnosis codes are clustered into 32 ADGs by the ACG software. All 32 ADGs, as well as sex and age categories, were included as independent variables within the logistic regression model. Age categories were based on ACG age categorization and clustered into 6 categories: aged 0 to 11 years, aged 12 to 34 years, aged 35 to 54 years, aged 55 to 69 years, aged 70 to 79 years, and 80 years or older.

Assessment of the model. The prediction model was assessed on both discriminatory and calibrating ability. To assess the discriminatory ability of the model, C statistic values were calculated in both the training and validation data sets. Values below 0.6 were considered poor; between 0.6 and 0.7, sufficient; between 0.7 and 0.8, good; between 0.8 and 0.9, very good; and above 0.9, excellent.13 The calibrating ability of the model was assessed by dividing the validation data set population into deciles based on ascending predicted values for being a complex patient. For each group, the mean observed and expected values were plotted against each other in the calibration plot. Models with a 45-degree angle plot (mean observed value equals mean expected value) were considered perfectly calibrated. Models below this reference line are overestimating, whereas models above it are underestimating.

RESULTS

Population Characteristics

Population characteristics were compared for both the training and validation data sets. Table 1 shows the differences in population characteristics between the 2 data sets. The 2 study populations had comparable mean ages of 40.8 and 40.6 years, respectively. The percentage of women was 51.4% and 50.1%, respectively, and the percentage of complex patients was comparable in both data sets at 0.9% and 0.8%.

The percentage of individuals with at least 2 acute care visits in 1 year was lower in the validation data set than in the training data set: 1.6% vs 2.7%. More health problems were observed in the validation data set. Somatic chronic diseases were more prevalent in the validation data set, at 46.8% compared with 39.8%. In addition, a higher prevalence was found of psychiatric health problems (40.6% vs 26.6%) and of social health problems (20.2% vs 10.9%). Common conditions such as depression, diabetes, hypertension, ischemic heart disease, asthma, and chronic obstructive pulmonary disease were comparably prevalent in both populations.

Model Performance

Odds ratios. Figure 1 provides an overview of the odds ratios (ORs) for each variable in the model. eAppendix B gives additional information about the ORs along with their 95% CIs.

Age and sex were not statistically significantly associated with the outcome. The following main predictors were identified: ADG categories chronic medical, stable (OR, 2.78; 95% CI, 2.35-3.29) and chronic medical, unstable (OR, 2.89; 95% CI, 2.49-3.35); psychosocial, stable (OR, 3.36; 95% CI, 2.92-3.86); and psychosocial, unstable (OR, 2.95; 95% CI, 2.33-3.73). In addition to the chronic medical and the psychosocial ADGs, the 3 ADGs time limited: major (OR, 2.37; 95% CI, 1.73-3.25), injuries/adverse effects: major (OR, 2.30; 95% CI, 1.89-2.79), and signs/symptoms: uncertain (OR, 2.95; 95% CI, 2.50-3.49) were also highly associated with the outcome.

Discriminatory ability. The discriminatory ability of the prediction model for identifying complex patients was assessed by calculation of C statistics. The C statistics, estimated for the training and validation data sets, are presented in Table 2.

Calibrating ability. Figure 2 shows the calibrating ability of the prediction model. The model is overestimating, meaning that it is identifying more complex patients than were observed in the study population. In eAppendix C, the observed number of complex patients, mean expected values, and standard errors are presented for each decile of population 2.

DISCUSSION

With this study, we have developed a prediction model for patients with complex care needs. Although a good discriminatory performance is shown, the calibrating ability is modest because more complex patients were identified based on the predicted values than were observed in reality. This study confirms results of previous studies, showing that the ACG System ADGs form a good tool to determine the level of multimorbidity,^14,15 which plays an important role in complex patient groups. With the highest weightings in the model for stable psychosocial problems, the proven prevalence of psychosocial conditions among our group of complex patients¹⁶ is taken into account. Stable and unstable chronic medical problems and unstable psychosocial problems are strongly associated with the outcome. Next to chronic medical and psychosocial problems, uncertain signs or symptoms also seem to have a high predictive effect on being a complex patient.

In our study, a statistics-based quantitative tool was found to identify patients with complex care needs. Because our model is overestimating complex patients, this tool based on routinely collected data should be used as a first screening. Additional qualitative screening of identified complex patients is of great importance to select the right subpopulation for interventions. Additional qualitative screening also allows for distinction between patients with high care needs that can be avoided with proactive care and patients in whom high care needs cannot be avoided. As our model was not built to distinguish between avoidable and unavoidable emergency care, this distinction with qualitative screening is of great importance. In addition, to investigate the practical use of our identification method, intervention studies with complex patients identified with our screening method should follow.

Further, high-quality registration is a necessary condition to be able to use complete health records for the identification of complex patients. Results of this study show that the prevalence of registered somatic chronic problems, as well as psychiatric and social health problems, differed between both study populations. These differences might be due to dissimilar registration policies rather than to prevalence differences. For efficient and practical PHM approaches using data-driven identification methods to designate resources to the patient groups in which they will be most beneficial, good quality of registration needs to be pursued. Once practitioners realize that registry data are reused for producing prediction models that are helpful in practice, improvement of registration habits is possible. In addition to the differences between the 2 study populations, the prevalence of social problems seemed lower than expected. Because studies have shown that complex patients are more likely to have underlying social problems, such as low income, living alone, living in a deprived area, and being less likely to own a home,^17-19 alongside chronic and psychosocial problems, the registration of social problems in primary care should be emphasized and stimulated. Alternatively, the identification of vulnerable and complex patient groups may be improved by creating access to social data sources. However, most risk stratification tools such as the ACG System do not currently include social data. In addition, linkage of social data to primary care data still entails significant information security issues.

An accurate identification of complex patients in primary care can be of great value to health care systems. Not only can early identification prevent deterioration of patients’ health, but health resources can also be put toward groups of patients who will benefit most from them. As general practitioners’ workload is increasing, resulting in rising pressure on them in most European countries,²⁰ allocation of available resources in primary care is of utmost importance. In situations in which different types of health care data can easily be combined, such as in managed care organizations, more complete profiles of patients can be used to allocate resources to the patient groups for whom they will be most beneficial. Risk stratification approaches to identify subpopulations have proven valuable in forms of both tailored care interventions and improved care management.^21,22 Effective PHM interventions following identification of complex groups of patients may lead to improved health outcomes, not only for this group, but for the total population.

Limitations

This research is based on a definition of complex patients that is mainly composed of biomedical characteristics. Because social determinants are underrepresented in primary care registration, defining complex patients was mostly restricted to the somatic physical and psychiatric diagnoses of patients. Although we believe that we have identified a group of complex patients with unmet health needs, the social health burden is most likely to be underrecorded. This may have caused our selected population of complex patients (outcome) to miss an important group of patients. As our PHM approach using primary care data is a holistic approach, the biomedical focus of registry systems may still cause important gaps in patients’ complete profiles.

Further, the age categories used in this study are broad. We explored the use of smaller age categories (eg, 5-year age bands) and the use of age as a continuous variable, but surprisingly age did not contribute statistically significantly to the identification of patients with complex care needs.

Lastly, by selecting people registered with one of the participating general practices for a period of 12 months, people born within this 12-month period were excluded from this study. However, very few infants would meet our requirements to be classified as a patient with complex care needs according to our definition.

CONCLUSIONS

Using broad morbidity groups of diagnoses or comorbidities (such as ADGs) seems to be an effective approach to identify complex patients in primary care. Risk stratification tools are key methods in putting available registry data to good use in order to identify complex patient groups. In addition to biomedical health determinants, social determinants play an important role in the identification of complex patients. Effective identification of complex patients in primary care can result in appropriate and proactive care management for this group of patients, benefiting the total primary care population.

Acknowledgments

The authors acknowledge Stichting Georganiseerde eerstelijnszorg Zoetermeer and Stichting Gezondheidscentra Nijkerk for providing research data, in particular data manager Carla van der Brugghen for providing supplemental information on the data and helpful suggestions for proper handling of the data, and the data protection officers for critical input on establishing a data privacy framework.

Author Affiliations: Department of Public Health and Primary Care, Leiden University Medical Centre (SAMG, JCV, SPS, MEN, MAB), the Hague, the Netherlands; Jan van Es Instituut (SAMG, MAB), Ede, the Netherlands; Mathematical Institute, Leiden University (MF), Leiden, the Netherlands; Medical Statistics Department of Biomedical Data Science, Leiden University Medical Center (MF), Leiden, the Netherlands; Princess Maxima Center for Pediatric Oncology (MF), Utrecht, the Netherlands; Department of Health Policy and Management, Bloomberg School of Public Health, Johns Hopkins University (SPS), Baltimore, MD.

Source of Funding: Noaber Foundation, Lunteren, the Netherlands.

Author Disclosures: Dr Sutch reports consultancies from Johns Hopkins. The remaining authors report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (SAMG, JCV, SPS, MAB); acquisition of data (SAMG, MAB); analysis and interpretation of data (SAMG, JCV, MF, SPS, MAB); drafting of the manuscript (SAMG, JCV, SPS, MAB); critical revision of the manuscript for important intellectual content (MF, SPS, MEN, MAB); statistical analysis (SAMG, JCV, MF, SPS, MAB); provision of patients or study materials (MAB); obtaining funding (MAB); administrative, technical, or logistic support (SAMG, MEN); and supervision (MF, MEN, MAB).

Address Correspondence to: Shelley-Ann M. Girwar, MSc, Department of Public Health and Primary Care, Leiden University Medical Centre, Turfmarkt 99, 2511 DP, the Hague, the Netherlands. Email: s.a.girwar@lumc.nl.

REFERENCES

1. Vogeli C, Shields AE, Lee TA, et al. Multiple chronic conditions: prevalence, health consequences, and implications for quality, care management, and costs. J Gen Intern Med. 2007;22(suppl 3):391-395. doi:10.1007/s11606-007-0322-1

2. Starfield B, Lemke KW, Bernhardt T, Foldes SS, Forrest CB, Weiner JP. Comorbidity: implications for the importance of primary care in ‘case’ management. Ann Fam Med. 2003;1(1):8-14. doi:10.1370/afm.1

3. Freund T, Mahler C, Erler A, et al. Identification of patients likely to benefit from care management programs. Am J Manag Care. 2011;17(5):345-352.

4. Gawande A. The hot spotters. The New Yorker. January 24, 2011:40-51.

5. Roblin DW, Segel JE, McCarthy RJ, Mendiratta N. Comparative effectiveness of a complex care program for high-cost/high-need patients: a retrospective cohort study. J Gen Intern Med. 2021;36(7):2021-2029. doi:10.1007/s11606-021-06676-x

6. Joynt KE, Gawande AA, Orav EJ, Jha AK. Contribution of preventable acute care spending to total spending for high-cost Medicare patients. JAMA. 2013;309(24):2572-2578. doi:10.1001/jama.2013.7103

7. Lee NS, Whitman N, Vakharia N, Taksler GB, Rothberg MB. High-cost patients: hot-spotters don’t explain the half of it. J Gen Intern Med. 2017;32(1):28-34. doi:10.1007/s11606-016-3790-3

8. Iovan S, Lantz PM, Allan K, Abir M. Interventions to decrease use in prehospital and emergency care settings among super-utilizers in the United States: a systematic review. Med Care Res Rev. 2020;77(2):99-111. doi:10.1177/1077558719845722

9. Forrest CB, Lemke KW, Bodycombe DP, Weiner JP. Medication, diagnostic, and cost information as predictors of high-risk patients in need of care management. Am J Manag Care. 2009;15(1):41-48.

10. Haas LR, Takahashi PY, Shah ND, et al. Risk-stratification methods for identifying patients for care coordination. Am J Manag Care. 2013;19(9):725-732.

11. Huntley AL, Johnson R, Purdy S, Valderas JM, Salisbury C. Measures of multimorbidity and morbidity burden for use in primary care and community settings: a systematic review and guide. Ann Fam Med. 2012;10(2):134-141. doi:10.1370/afm.1363

12. Lemke KW, Weiner JP, Clark JM. Development and validation of a model for predicting inpatient hospitalization. Med Care. 2012;50(2):131-139. doi:10.1097/MLR.0b013e3182353ceb

13. Šimundic AM. Measures of diagnostic accuracy: basic definitions. EJIFCC. 2009;19(4):203-211.

14. Sibley LM, Moineddin R, Agha MM, Glazier RH. Risk adjustment using administrative data-based and survey-derived methods for explaining physician utilization. Med Care. 2010;48(2):175-182. doi:10.1097/MLR.0b013e3181c16102

15. Starfield B, Weiner J, Mumford L, Steinwachs D. Ambulatory care groups: a categorization of diagnoses for research and management. Health Serv Res. 1991;26(1):53-74.

16. Mautner DB, Pang H, Brenner JC, et al. Generating hypotheses about care needs of high utilizers: lessons from patient interviews. Popul Health Manag. 2013;16(suppl 1):S26-S33. doi:10.1089/pop.2013.0033

17. Colligan EM, Pines JM, Colantuoni E, Wolff JL. Factors associated with frequent emergency department use in the Medicare population. Med Care Res Rev. 2017;74(3):311-327. doi:10.1177/1077558716641826

18. Fitzpatrick T, Rosella LC, Calzavara A, et al. Looking beyond income and education: socioeconomic status gradients among future high-cost users of health care. Am J Prev Med. 2015;49(2):161-171. doi:10.1016/j.amepre.2015.02.018

19. Krieg C, Hudon C, Chouinard MC, Dufour I. Individual predictors of frequent emergency department use: a scoping review. BMC Health Serv Res. 2016;16(1):594. doi:10.1186/s12913-016-1852-1

20. Schäfer WLA, Boerma WGW, Spreeuwenberg P, Schellevis FG, Groenewegen PP. Two decades of change in European general practice service profiles: conditions associated with the developments in 28 countries between 1993 and 2012. Scand J Prim Health Care. 2016;34(1):97-110. doi:10.3109/02813432.2015.1132887

21. Burton LC, Skinner EA, Uscher-Pines L, et al. Health of Medicare Advantage plan enrollees at 1 year after Hurricane Katrina. Am J Manag Care. 2009;15(1):13-22.

22. Soto-Gordoa M, de Manuel E, Fullaondo A, et al. Impact of stratification on the effectiveness of a comprehensive patient-centered strategy for multimorbid patients. Health Serv Res. 2019;54(2):466-473. doi:10.1111/1475-6773.13094