Application of New Method for Evaluating Performance of Fracture Risk Tool

TO THE EDITORS:

We previously published our experience with the Fracture Risk Calculator (FRC; Foundation for Osteoporosis Research and Education) within Kaiser Permanente Northern California.¹ Among nearly 95,000 female health plan members 50 years and older who underwent bone mineral density during 1997 to 2003, we compared the 10-year risk of hip fracture estimated using FRC with observed hip fracture outcomes.¹ A part of that analysis compared FRC performance with and without input of femoral neck bone mineral density (BMD). Based on area under the receiver operator characteristic curve (AUROC) methodology (c-statistic), we concluded that BMD did not enhance the tool’s performance.

While the AUROC is a widely used method to optimize sensitivity and specificity, other methods may also be used to evaluate risk-assessment tools. One such method is the net reclassification improvement (NRI).^2,3 The NRI calculation adds rates of appropriate reclassification of individuals to risk categories (to higher risk category for those who subsequently fracture and to lower risk category for those who do not) and subtracts inappropriate reclassifications; the result is a summative overall reclassifi cation improvement score ranging from —100% to +100%.²

We recently applied this methodology to examine the performance of FRAX, another web-based fracture—risk assessment tool developed by the World Health Organization.⁴ We found, as with FRC, that AUROC (c-statistic) was not improved with the addition of BMD to the FRAX model.⁴ However, the addition of BMD to FRAX did result in a significant positive NRI (5.5%, P <.01), using a cutoff derived from the maximum Youden’s J statistic (this identifies the threshold for optimal sensitivity and specifiity).⁴ We now similarly examine the addition of BMD to the FRC tool and its effect on the NRI using the same methodology.

The Youden’s J statistic identified a threshold of 1.3% for the FRC tool. When using 1.3% as a risk category threshold for assignment to higher versus lower risk,⁴ we found that the addition of BMD to FRC yielded a statistically non-significant NRI of 0.8% (P = .3). These analyses provide further evidence that the FRC tool’s performance appears to be minimally affected by the addition of BMD.

As methods for assessing risk prediction tools continue to be refined, further studies in larger and more diverse cohorts should be conducted to determine whether the addition of BMD provides improved discrimination in specific population subsets. In the meantime, our findings support the use of the FRC in populations where BMD is not known. Using limited electronic medical record data, this tool can identify higher-risk subgroups for further evaluation and management.

Erica M. Sobel, DOBruce Ettinger, MDJoan C. Lo, MDAlice R. Pressman, PhDAuthor Affiliations: From Department of Medicine (EMS, JCL), Kaiser Permanente Oakland Medical Center; Division of Research (BE, JCL, ARP), Kaiser Permanente Northern California, Oakland, CA.

Authorship Information: Concept and design (JCL, ARP); acquisition of data (JCL, ARP); analysis and interpretation of data (EMS, BE, JCL, ARP); drafting of the manuscript (EMS, BE, JCL); critical revision of the manuscript for important intellectual content (EMS, BE, JCL, ARP); and statistical analysis (EMS, ARP).

Author Disclosures: Dr Ettinger reports that he is a board member of the Foundation for Osteoporosis Research and Education. The other authors (EMS, JCL, ARP) report no relationship or financial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Address correspondence to: Joan C. Lo, MD, Kaiser Permanente Division of Research, 2000 Broadway, Oakland, CA 94612. E-mail: Joan.C.Lo@kp.org.1. Lo JC, Pressman AR, Chandra M, Ettinger B. Fracture risk tool validation in an integrated healthcare delivery system. Am J Manag Care. 2011;17(3):188-194.

2. Pencina MJ, D’Agostino RB Sr, Steyerberg EW. Extensions of net reclassification improvement calculations to measure usefulness of new biomarkers. Stat Med. 2011;30(1):11-21.

3. Pencina MJ, D’Agostino RB Sr, D’Agostino RB Jr, Vasan RS. Evaluating the added predictive ability of a new marker: from area under the ROC curve to reclassification and beyond. Stat Med. 2008;27(2):157-172; discussion 207-112.

4. Pressman AR, Lo JC, Chandra M, Ettinger B. Method for assessing fracture risk prediction models: experience with FRAX in a large integrated health care delivery system. J Clin Densitom. 2011;14(4):407-415.

Download PDF: Application of New Method for Evaluating Performance of Fracture Risk Tool