One Size Does Not Always Fit All in Value Assessment

Am J Manag Care. 2019;25(11):540-542Rising healthcare costs have led to the emergence of a host of value frameworks aimed at both defining and quantifying what value means for healthcare in the United States.¹ Healthcare organizations, patient advocacy groups, and think tanks across the country have developed such frameworks to assess the potential value of new therapies.^2,3 In the United States, the framework to address new drug evaluation and pricing developed by the Institute for Clinical and Economic Review (ICER) has caught the attention of private payers.⁴ Most recently, CVS Caremark announced plans to use the results from ICER’s cost-effectiveness assessments to guide formulary decision making, which could lead to the exclusion of some high-cost drugs from some of its plans.⁵

The ICER perspective on what value means in healthcare—and some of the core methodology that it uses to evaluate alternative technologies—is based on long-standing academic concepts about cost-effectiveness analyses. These have been used in decision making outside the United States, notably by the likes of the National Institute for Health and Care Excellence in England and Wales and the Pharmaceutical Benefits Advisory Committee in Australia. However, these approaches have faced criticism,⁶ not least because of the lack of attention given to heterogeneity in relative effectiveness and cost-effectiveness according to patients’ characteristics and preferences.⁷

The Second Panel on Cost-Effectiveness in Health and Medicine called for heterogeneity to be considered through the presentation of subgroup-specific cost-effectiveness, where appropriate evidence exists.⁸ Yet comparative and cost-effectiveness analyses have been slow to recognize heterogeneity and tend not to present subgroup value estimates.⁹ By focusing on evaluating the overall average effectiveness, these value frameworks do not encourage the generation of useful evidence on heterogeneity that can inform differential decisions about the extent to which particular subgroups may benefit from new, high-cost healthcare technologies.

In most published value assessments, globally, heterogeneity has not been featured strongly in the reports of the main clinical results, and in the cost-effectiveness analysis these are addressed post hoc, after the main model has been built. For example, ICER’s Evidence Rating Matrix¹⁰ makes no mention of whether a study attempts to detect or understand heterogeneity or report results by subgroup. There are genuine reasons to ignore heterogeneity in the absence of evidence, while there are cases where heterogeneity is ignored even with reliable evidence. ICER reports highlight both such cases.

One example in which evidence on heterogeneity could have been incorporated was ICER’s report on treatments for rheumatoid arthritis (RA). ICER stated, “RA remains a remarkably complex disease to diagnose and manage. There are multiple phenotypic and genotypic variations in the pathogenesis of the disease that affect both the course of RA and the outcome of therapy.”¹¹ Still, no attempt was made to evaluate cost-effectiveness of different therapeutic agents for subgroups. It is important to note that there was no direct evidence about treatment-effect heterogeneity across subgroups in any of the trials that were identified for the report. However, evidence beyond those trials clearly suggested that for patients receiving the control regimen, clinical responses differed according to age and functional status.¹² Hence, even if the relative effect of a new targeted immune modulator was constant across subgroups, there could still be substantial variation in the absolute effect scale required for estimates of cost-effectiveness.¹³ It is not clear how incorporating such heterogeneity might have changed the overall assessment, but at the least, it could have triggered a different conversation around value for certain groups of patients. More generally, ignoring heterogeneity could result in therapies that may be highly effective and cost-effective for one particular group of patients not receiving coverage and reimbursement because they are not cost-effective for everyone.

In contrast, when evaluating programmed cell death 1 receptor agents in the treatment of non—small cell lung cancer, ICER’s analyses relied on phase 2 and 3 trials that often did not have the power to establish subgroup effects reliably.¹⁴ Therefore, despite emerging practice-based evidence that testing the level of programmed cell death ligand 1 protein that a tumor expresses can significantly help determine which patients may benefit from treatment, there was no reliable evidence during the clinical trial stage of development to model treatment-effect heterogeneity and report subgroup analyses.

These cases demonstrate 2 key barriers to driving greater reflection of heterogeneity in policy choices: positioning and availability of sources. The first case shows that even when clear evidence of heterogeneity of effect is present in published evidence, it is not moved to the front of the conversation, perhaps because it was not directly studied in the regulatory trial contexts. The second example points to the limitations when there is a distinct lack of strong empirical evidence on heterogeneity at the time clinical trials are conducted, but such evidence does emerge in clinical practice.

Implications of the Failure to Account for Heterogeneity in Value Assessment

It is imperative that value assessments encourage the recognition of evidence on heterogeneity for 2 reasons. First, it is well established that generating and reporting differential value assessment across subgroups leads to substantial health gains, both through treatment selection and coverage.^15-17 This means that simply providing value assessments for overall populations—even when clinical evidence shows differential effectiveness across subpopulations—leads to a disconnect between the assessment of evidence by payers versus clinicians and patients. This disconnect can ultimately lead to inefficient decision making around reimbursement and pricing.

Second, the recognition of clinical effect heterogeneity in value assessments can incentivize the production of better evidence on heterogeneity in the future. Greater availability of such evidence is critical to optimizing the benefits of a technology in the population. The current paucity of evidence around heterogeneity in the effectiveness of new treatments could reflect the lack of incentives to generate this information for regulatory purposes. By honoring and highlighting evidence on heterogeneity, value assessments could change that narrative and promote the generation of evidence.

Strategies for Accounting for Heterogeneity

Subgroup-specific value assessment may be an efficient strategy for accounting for heterogeneity on costs and benefits. Some value assessment framework developers have been reluctant to consider analysis of cost-effectiveness by subgroup or individual, largely for either practical reasons (eg, the concern that such granularity in results will require far larger studies) or statistical reasons (eg, potential selection bias, concerns about the greater risk of false positives or the relaxing of the standards of certainty). However, there are a growing number of approaches geared to overcoming these concerns that have been investigated with some success in recent years.

For example, the use of instrumental variables to define individual-level treatment effects of various approaches to treat prostate cancer used the same data sets that were used to produce the populationwide estimates of effectiveness that determined policy in the United States.¹⁸ There have also been studies looking at various approaches to the use of propensity score matching to minimize selection bias in estimating cost-effectiveness by subgroups in the treatment of sepsis in the United Kingdom.¹⁹ Bayesian modeling approaches, in which multiple sources of data on the relationship between the characteristics of patients and their risks are considered, have also been suggested for more effective interpretation of potential subgroup effects.^19,20

These approaches have all shown how value assessment can be conducted in a way that better accounts for heterogeneity of treatment effect, and they highlight the need for a more nuanced view of the evidence hierarchy—one that recognizes a greater role for real-world data as a complement to traditional randomized controlled trial designs. The practice of cost-effectiveness analysis is mostly focused on the simple comparison of population-based treatment impacts. However, decision makers may benefit from knowing how these impacts vary across subsets of the population so that benefit designs or coverage decisions are aligned in an increasingly complex healthcare delivery system that is rapidly evolving toward an increased use of personalized medicine. One step forward in this evolution may occur if value assessments open a conversation about evidence of heterogeneity of effect with manufacturers during their initial interactions around sharing public and proprietary evidence on new drugs. No one is recommending that a health technology assessment body make statements based on nonexistent evidence, but where it exists, it should not be ignored or made a footnote; patients deserve better.

Conclusions

Laying a clear path for incorporating reliable evidence on heterogeneity in value assessments could improve its applicability for healthcare decision making. This could include not reporting population average cost-effectiveness results when there are distinct differences in subgroup-specific results and sufficient accounting for such heterogeneity among patients. Importantly, creating an environment that respects and rewards evidence on heterogeneity should help value frameworks evolve to become more applicable and appropriate for payers’ decisions and promote generation of evidence on heterogeneity.

The worlds of comparative effectiveness and cost-effectiveness research must catch up with the evolution of how healthcare is shifting its emphasis from addressing disease in populations to addressing disease in patients, both now and in the future. There is no better time to begin realizing this change in perspective than now.Author Affiliations: The CHOICE Institute, Departments of Pharmacy, Health Services, and Economics, University of Washington (AB), Seattle, WA; London School of Hygiene & Tropical Medicine (RG), London, United Kingdom; Personalized Medicine Coalition (DP), Washington, DC; Parexel International (WS), Durham, NC.

Source of Funding: Dr Stevens received funding from Pharmaceutical Research and Manufacturers of America for devising, drafting, and reviewing the paper.

Author Disclosures: Dr Basu served as a consultant with Paraxel International for this work and has also consulted with Salutis Consulting LLC. Dr Stevens receives funding from life science companies for consultancy services and was paid to help write this commentary. Drs Grieve and Pritchard report no relationship or financial interest that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design (AB, RG, DP, WS); analysis and interpretation of data (AB); drafting of the manuscript (AB, RG, DP, WS); critical revision of the manuscript for important intellectual content (AB, RG, DP, WS); provision of patients or study materials (DP); and obtaining funding (WS).

Address Correspondence to: Warren Stevens, PhD, Parexel International, 2520 Meridian Pkwy, Ste 200, Durham, NC 27713. Email: wstevens15@gmail.com.REFERENCES

1. Kesselheim AS, Avorn J, Sarpatwari A. The high cost of prescription drugs in the United States: origins and prospects for reform. JAMA. 2016;316(8):858-871. doi: 10.1001/jama.2016.11237.

2. Wilson L, Lin T, Wang L, et al. Evaluation of the ASCO value framework for anticancer drugs at an academic medical center. J Manag Care Spec Pharm. 2017;23(2):163-169. doi: 10.18553/jmcp.2017.23.2.163.

3. Schnipper LE, Davidson NE, Wollins DS, et al; American Society of Clinical Oncology. American Society of Clinical Oncology statement: a conceptual framework to assess the value of cancer treatment options. J Clin Oncol. 2015;33(23):2563-2577. doi: 10.1200/JCO.2015.61.6706.

4. ICER launches new drug assessment program with $5.2M from Arnold Foundation [news release]. Houston, TX, and Boston, MA: Arnold Foundation; July 21, 2015. arnoldventures.org/newsroom/icer-launches-new-drug-assessment-program-with-5-2-million-award-from-the-laura-and-john-arnold-foundation. Accessed September 16, 2019.

5. Current and new approaches to making drugs more affordable. CVS Health website. cvshealth.com/sites/default/files/cvs-health-current-and-new-approaches-to-making-drugs-more-affordable.pdf. Published August 2018. Accessed September 16, 2019.

6. Birch S, Gafni A. On being NICE in the UK: guidelines for technology appraisal for the NHS in England and Wales. Health Econ. 2002;11(3):185-191.

7. Espinoza MA, Manca A, Claxton K, Sculpher MJ. The value of heterogeneity for cost-effectiveness subgroup analysis: conceptual framework and application. Med Decis Making. 2014;34(8):951-964. doi: 10.1177/0272989X14538705.

8. Sanders GD, Neumann PJ, Basu A, et al. Recommendations for conduct, methodological practices, and reporting of cost-effectiveness analyses: Second Panel on Cost-Effectiveness in Health and Medicine [erratum in JAMA. 2016;316(18):1924. doi: 10.1001/jama.2016.15518]. JAMA. 2016;316(10):1093-1103. doi: 10.1001/jama.2016.12195.

9. Kravitz RL, Duan N, Braslow J. Evidence-based medicine, heterogeneity of treatment effects, and the trouble with averages [erratum in Milbank Q. 2006;84(4):759-760]. Milbank Q. 2004;82(4):661-687. doi: 10.1111/j.0887-378X.2004.00327.x.

10. Ollendorf D, Pearson SD. ICER evidence rating matrix: a user’s guide. Institute for Clinical and Economic Review website. icer-review.org/wp-content/uploads/2013/04/Rating-Matrix-User-Guide-Exec-Summ-FINAL.pdf. Published 2013. Accessed September 16, 2019.

11. Targeted immune modulators for rheumatoid arthritis: effectiveness & value. Institute for Clinical and Economic Review website. icer-review.org/wp-content/uploads/2016/08/NE_CEPAC_RA_Evidence_Report_FINAL_040717.pdf. Published April 7, 2017. Accessed September 16, 2019.

12. Hetland ML, Christensen IJ, Tarp U, et al; All Departments of Rheumatology in Denmark. Direct comparison of treatment responses, remission rates, and drug adherence in patients with rheumatoid arthritis treated with adalimumab, etanercept, or infliximab: results from eight years of surveillance of clinical practice in the nationwide Danish DANBIO registry. Arthritis Rheum. 2010;62(1):22-32. doi: 10.1002/art.27227.

13. Kent DM, Nelson J, Dahabreh IJ, Rothwell PM, Altman DG, Hayward RA. Risk and treatment effect heterogeneity: re-analysis of individual participant data from 32 large clinical trials. Int J Epidemiol. 2016;45(6):2075-2088. doi: 10.1093/ije/dyw118.

14. Treatment options for advanced non-small cell lung cancer: effectiveness, value and value-based price benchmarks. Institute for Clinical and Economic Review website. icer-review.org/wp-content/uploads/2016/08/MWCEPAC_NSCLC_Evidence_Report_Plus_Supplement_101716.pdf. Updated October 7, 2016. Accessed September 16, 2019.

15. Basu A. Economics of individualization in comparative effectiveness research and a basis for a patient-centered health care. J Health Econ. 2011;30(3):549-559. doi: 10.1016/j.jhealeco.2011.03.004.

16. Basu A, Jena AB, Philipson TJ. The impact of comparative effectiveness research on health and health care spending. J Health Econ. 2011;30(4):695-706. doi: 10.1016/j.jhealeco.2011.05.012.

17. Espinoza MA, Manca A, Claxton K, Sculpher MJ. The value of heterogeneity for cost-effectiveness subgroup analysis: conceptual framework and application. Med Decis Making. 2014;34(8):951-964. doi: 10.1177/0272989X14538705.

18. Basu A. Estimating person-centered treatment (PeT) effects using instrumental variables: an application to evaluating prostate cancer treatments. J Appl Econ (Chichester Engl). 2014;29(4):671-691. doi: 10.1002/jae.2343.

19. Kreif N, Grieve R, Radice R, Sadique Z, Ramsahai R, Sekhon JS. Methods for estimating subgroup effects in cost-effectiveness analyses that use observational data. Med Decis Making. 2012;32(6):750-763. doi: 10.1177/0272989X12448929.

20. Heckman JJ, Lopes HF, Piatek R. Treatment effects: a Bayesian perspective. Econom Rev. 2014;33(1-4). doi: 10.1080/07474938.2013.807103.

Download PDF: One Size Does Not Always Fit All in Value Assessment