Expanding Use of Multi-Criteria Decision Analysis for Health Technology Assessment

The Opportunities and Challenges of Rethinking Our Approach to Value Assessment in Healthcare

As America’s healthcare system continues to evolve, it is critical that our perception of care and its value to patients evolve with it. In the past, value assessments have marginalized patients’ perspectives in favor of other, more easily quantifiable variables. Unfortunately, this approach to value assessment hasn’t been able to capture individual health states or preferences because it fails to engage with the most important stakeholders: the individuals receiving the care.

Take, for example, the quality-adjusted life-year, also known as the QALY. Explained at the most basic level, the QALY is a measurement of how an intervention improves a patient’s quality and quantity of life. The QALY aims to encapsulate the value of healthcare interventions in a single index number, where 1 equates to 1  year of perfect health and 0 is associated with death.

From the patient perspective, assessing the value and impact of care through a summary metric is akin to summarizing a 200-page novel in a single word. Although many experts acknowledge the limitations of the QALY metric, they often throw their hands up and assert that patient perspectives are just too difficult to quantify as a practicable metric.

But things are beginning to change.

This year, health economists and health services researchers rolled up their sleeves to offer alternative approaches to measure value as part of the Pharmaceutical Research and Manufacturers of America Foundation’s 2019 Challenge Awards. The awards presented researchers with a single prompt:

“What are innovative, patient-centered approaches to contribute to healthcare value assessment that move beyond the inherent limitations of analyses based on the quality-adjusted life-year metric?”

Researchers responded with myriad novel, innovative, and practical approaches to value assessment that enhance or mitigate past the QALY and allow deep engagement with patients. Perhaps more important, the volume of substantive submissions undermined the idea that successfully incorporating the patient voice into healthcare assessments was too difficult.

Of all the approaches offered, 4 winning submissions were selected based on their innovative and pragmatic approaches to value assessment. Although each of the approaches differs in methodology and design, a common theme throughout is the realistic way in which they account for the perspectives of patients. In one selected model, for example, the authors propose to inform value assessment with learnings from patient-focused drug development (PFDD), a program instituted by the FDA to capture the experiences, perspectives, and needs of patients based on their unique symptoms and medical histories. Another approach argues for using patient stakeholder groups and existing patient registries to better incorporate patient perspectives.

One of the submissions takes value assessment a step further, arguing that conventional approaches to value measurement that emphasize the average (mean) of healthcare outcomes are flawed. According to the authors, because patient care is about more than just average outcomes, we should incorporate the statistical variance and skewness of treatment outcome distributions to more effectively capture the wide spectrum of patient experiences.

Interestingly, many of the selected winners offer methodologies that aren’t constructed from scratch. Instead, they expand on existing infrastructure, such as the PFDD, which is already purposed to capture patient perspectives on a medicine’s effectiveness. Another winning submission discusses the advantages of applying multiple-criteria decision analysis—a long-standing decision-making tool used in other industries—given its ability to capture multiple dimensions of value that often go unmeasured.

But beyond just the winning approaches, many, if not most, of the submissions present realistic methods that expand the scope of value assessment, engage with patients, and innovate beyond the QALY.

It is clear that ongoing efforts such as the Challenge Awards that seek to move beyond the QALY are making meaningful headway in improving value assessments for patients. We should applaud the worthy goals of these efforts to fundamentally rethink what constitutes value.

Eileen Cannon

President

Pharmaceutical Research and Manufacturers of America Foundation

Expanding Use of Multi-Criteria Decision Analysis for Health Technology Assessment

Charles E. Phelps, PhD

INTRODUCTION

Cost-effectiveness analysis (CEA), founded on basic economic principles, remains the de facto standard for health technology assessment (HTA).¹ Properly done, it measures efficiency of medical interventions across broad illness categories. Unfortunately, while accurate within its defined structure, CEA is incomplete.² A recent review lists a dozen potential factors commonly omitted from CEA.³ As CEA’s use expands, better methods to include these multiple dimensions become increasingly important.

MULTI-CRITERIA DECISION ANALYSIS

Decision support tools that allow multiple dimensions of value have existed for decades, with relatively little use in HTA until recently.⁴ One website lists more than 2 dozen software implementations of various multi-criteria decision analysis (MCDA) models.⁵ While differing in details, most MDCA models have several elements in common:

• They formally incorporate multiple dimensions of value.
• Decision makers select relevant value dimensions and specify their relative importance (“weights”). Processes to elicit decision makers’ weights vary considerably across models.
• They combine each candidate’s performance along chosen value dimensions into comprehensive scores that are used to rank candidates.
• Despite differences in intellectual heritage, these value metrics often use simple linear combinations (using decision makers’ weights) of each candidate’s performance on each value dimension.

MCDA’s merits are well known. They include transparency, “flight simulator” testing, guiding data improvements, decision convergence, and avoidance of many cognitive errors associated with intuitive decision making⁶ that the field of behavioral economics has established.⁷

MCDA has another healthcare-related virtue: The same general models can guide decisions at multiple levels, beginning with, for example, individual patients choosing among various treatment options, and continuing to more aggregated decision levels such as deciding on technology investments and health plan coverage. Biopharmaceutical manufacturers can use MCDA to prioritize among potential research and development investments. Further, MCDA can assist in research funding allocation (eg, National Institutes of Health or private foundations).

In each setting, “decision makers” must choose which attributes (dimensions of value) to include in decision models and specify importance-weights for each attribute. In some cases, they must also score candidates on subjective attributes, such as “improves equity” or “possible scientific breakthrough,” for which no specific measures exist. Learning how to best accomplish these tasks in different settings—from individual patients’ choices to system-wide investment decisions—stands as the most important barrier to MCDA’s widespread use to guide healthcare decisions. We simply do not have enough experience with these models, in either individual (patients) or group (organizational) decisions, to know how to maximize their usability and hence their use. In group settings, this interacts heavily with the choice of voting methods, which can alter decision outcomes.

BARRIERS TO USE

During extensive field testing and international presentations of a sophisticated MCDA model to prioritize vaccine development and use, Phelps and Madhavan compiled common concerns about MCDA⁸:

•  “MCDA requires too much data.”

Response: The problems, not the MCDA models, create this complexity.

•  “It’s easy to ‘game’ the results.”

Response: MCDA models actually make it more difficult to do this, since the decision structures (weights on attributes) are wholly visible.

•  “MCDA models are too complicated to use.”

Response: With individual patient decision making, this is clearly not true, since they have been used to assist patient decision making for decades (Panattoni L, Phelps CE, Lieu TA, et al, unpublished data, 2019).⁹ In group settings, much research remains to be completed to establish best approaches to combining individual into group preferences.

•  “You can’t use MCDA in situations with a budget constraint.”

Response: While once true, this criticism no longer applies.¹⁰

ENHANCING USABILITY

Some aspects of MCDA affect potential users at all levels. Key issues include: (1) choosing which attributes to include, (2) elicitation of decision weights, and (3) scoring each candidate on each attribute dimension. Choosing among MCDA models requires consideration of user friendliness and of voting methods to determine group choices.

1. User Friendliness

Various MCDA models differ in demands on decision makers, which is important both for individual patients and policy-setting groups. A review of available models suggests that multi-attribute utility theory (MAUT) creates fewest demands on users, while analytic hierarchy process (AHP) creates the greatest. Determining decision makers’ weights, for K choices, AHP typically has K-1 times the number of decisions as MAUT. Scoring candidates on N subjective attributes, AHP requires N times as many decisions as MAUT. Other MCDA software generally falls between these 2.

2. Voting Methods

Various voting methods differ greatly in expressivity. Six well-known ballot types allow creation of rank order lists, essential in many MCDA processes. The Table shows the general formulas and the number of different possible expressions allowable for 5 candidates with 6 different voting methods. Huge differences in potential expressiveness are obvious. Among these, cumulative voting and range voting allow direct determination of value weights in group voting settings.

For comparison, 6-month-old infants understand about 6 words, and 18-month-old infants about 50. Dogs’ word comprehension ranges between 150 and 400 words. Six-year-old children master about 2500 words and average adults about 20,000 to 35,000 words. Simple considerations suggest using majority judgment, cumulative voting, or range voting when detailed understanding of voters’ preferences is desirable. Common ballot forms “dumb down” vocabularies to those of infants or dogs.

CONCLUDING COMMENTS

Expanding the understanding and measurement of value beyond CEA requires a systems analysis perspective. Control of complex systems requires the ability to measure and combine information in new ways, a task for which MCDA is ideally suited.¹¹

Getting “there” requires systematic accumulation of data that do not currently exist.¹² Data-gathering efforts will have better focus if consensus emerges regarding core attributes for MCDA model use.¹³ Software to support MCDA must become much friendlier for group decision making than current offerings, most importantly by introducing easy methods to elicit group preferences. Similar efforts to simplify methods for individual patient use are central to improved adoption of MCDA in clinical settings for truly “personalized” medicine. Human factors loom large in these issues. Graduate programs training new healthcare professionals should expand cost-effectiveness and decision analysis courses to include MCDA methods.¹⁴

Ultimately, MCDA use will expand with improved usability and familiarity. People resist new ideas even when old ones are insufficient. Buckminster Fuller noted, “You never change things by fighting against the existing reality. To change something, build a new model that makes the old model obsolete.” Therein lies the challenge. We cannot wait.

Dr Phelps is provost emeritus of the University of Rochester.

Acknowledgment: Dr Phelps acknowledges the extensive contributions of Guru Madhavan, PhD, Norman R. Augustine Senior Scholar and director of programs at the National Academy of Engineering, through previous work that helped build the foundation for this analysis.

1. Garber AM, Phelps CE. Economic foundations of cost-effectiveness analysis. J Health Econ. 1997;16(1):1-31. doi: 10.1016/s0167-6296(96)00506-1.

2. Phelps C, Madhavan G, Rappuoli R, Colwell R, Fineberg H. Beyond cost-effectiveness: using systems analysis for infectious disease preparedness. Vaccine. 2017;35(suppl 1):A46-A49. doi: 10.1016/j.vaccine.2016.08.090.

3. Lakdawalla DN, Doshi JA, Garrison LP Jr, Phelps CE, Basu A, Danzon PM. Defining elements of value in health care—a health economics approach: an ISPOR Special Task Force report [3]. Value Health. 2018;21(2):131-139. doi: 10.1016/j.jval.2017.12.007.

4. Köksalan M, Wallenius J, Zionts S. Multiple Criteria Decision Making: From Early History to the 21st Century. Singapore: World Scientific; 2011.

5. Software related to MCDM. International Society on Multiple Criteria Decision Making website. mcdmsociety.org/content/software-related-mcdm. Accessed April 20, 2019.

6. Phelps CE, Madhavan G. Using multicriteria approaches to assess the value of health care. Value Health. 2017;20(2):251-255. doi: 10.1016/j.jval.2016.11.011.

7. Kahneman D. Thinking, Fast and Slow. New York, NY: Farrar, Straus and Giroux; 2011.

8. Phelps CE, Madhavan G. Valuing health: evolution, revolution, resistance, and reform. Value Health. 2019;22(5):505-510. doi: 10.1016/j.jval.2019.01.010.

9. Dolan JG. The potential impact of a decision support system based on the analytical hierarchy process (AHP) on physicians decisions regarding antibiotic-treatment. Clin Res. 1988;36(3):A710.

10. Phelps C, Madhavan G. Resource allocation in decision support frameworks. Cost Eff Resour Alloc. 2018;16(suppl 1):48. doi: 10.1186/s12962-018-0128-5.

11. Madhavan G, Phelps CE, Rouse WB, Rappuoli R. Vision for a systems architecture to integrate and transform population health. Proc Nat Acad Sci U S A. 2018;115(50):12595-12602. doi: 10.1073/pnas.1809919115.

12. Madhavan G, Phelps C, Sangha K, Levin S, Rappuoli R. Bridging the gap: need for a data repository to support vaccine prioritization efforts. Vaccine. 2015;33(suppl 2):B34-B39. doi: 10.1016/j.vaccine.2015.02.032.

13. Garrison LP Jr, Neumann PJ, Willke RJ, et al. A health economics approach to US value assessment frameworks—summary and recommendations of the ISPOR Special Task Force report [7]. Value Health. 2018;21(2):161-165. doi: 10.1016/j.jval.2017.12.009.

14. Phelps C, Madhavan G, Rappuoli R, Levin S, Shortliffe E, Colwell R. Strategic planning in population health and public health practice: a call to action for higher education. Milbank Q. 2016;94(1):109-125. doi: 10.1111/1468-0009.12182.

Download PDF: Expanding Use of Multi-Criteria Decision Analysis for Health Technology Assessment