Using Guidelines: When Is It Appropriate?

ABSTRACT

Guidelines are concise, usable messages that are intended to influence behaviors. This article aims to review guideline development methodology to inform guide­line users, decision makers, and patients regarding key aspects to consider when both appraising the quality of this important knowledge tool and when judging if a practice should be adapted or changed based on the recommendations. Given the potential guideline benefits and harms, it is of primary importance to ensure a rigorous development process. Moreover, guideline users, decision makers, and patients should be able to evaluate the following: if the guideline is needed; if the scope, analytical framework, and key questions are relevant to their clinical reality; if the recommendations are applicable; and if the process of recommen­dations development is reliable and valid. Appraising guidelines will allow those key stakeholders to judge when a recommendation is applicable or not to their specific circumstances in order to avoid an inappropriate use.

Graham et al¹ developed the knowledge-to-action pro­cess to represent how knowledge is created, dissemi­nated, and implemented to improve the quality of care. Knowledge creation comprises knowledge inquiry (eg, primary studies), knowledge synthesis, and knowledge tools—such as guidelines²—which are concise and usable messages that are in­tended to influence behaviors.

Guidelines are defined as, “systematically developed state­ments to assist practitioner and patient decisions about appro­priate healthcare for specific clinical circumstances.”³ Guidelines are developed with the intent to improve the quality of care and outcomes, reduce inappropriate variation in practice, summarize evidence, favor an efficient resource use, make clinical decisions more transparent, identify areas of knowledge for which research should be prioritized, empower patients, inform public policy, and support quality control activities.²

This article aims to review guideline development methodol­ogy to inform users, decision makers, and patients regarding key aspects to consider both when appraising the quality of this im­portant knowledge tool and when judging if a practice should be adapted or changed based on the recommendations. After outlin­ing the potential benefits and risks of guidelines, aspects related to appropriate guideline development and use will be detailed.

Potential Benefits and Harms of Guidelines

Woolf et al⁴ identified the potential benefits of guidelines for patients, healthcare professionals, medical researchers, and healthcare systems. For patients, one important benefit is the promotion of therapies that have the potential for better out­comes; clear guidelines can also improve consistency of care. For healthcare professionals, guidelines clarify which interventions are better supported by the evidence, help to explain the risks and benefits to the healthcare consumer, and can constitute a point of reference for quality improvement activities. For medi­cal researchers, guidelines can emphasize gaps in knowledge. Ad­ditionally, healthcare systems may also benefit from guidelines that highlight cost-effective interventions.

In spite of the numerous benefits, if evidence is misleading or misinterpreted, or if the patients’ needs are not prioritized, guidelines have the potential to cause harm as well.⁴ First, pa­tients can be negatively impacted if guidelines do not allow cli­nicians to tailor care to specific circumstances. Guidelines are also influenced by the opinions and experiences of development workgroup members, which can have an impact on the priori­ties being considered. For example, members may be distracted from focusing on patients’ needs if they prefer to build their recommendations based on costs. Second, healthcare profession­als can be affected if the recommendations are wrong, outdat­ed, or contradict other recommendations, which might result in the unfair judgment of clinicians. Third, if further research is inappropriately discouraged, this may negatively impact medical researchers. Finally, the healthcare system may also be compro­mised if guideline implementation generates a waste of resources by recommending unnecessary interventions.

Appropriate Guideline Development

Recognizing the potential for harm stresses the importance of following a rigorous, transparent, and credible guideline develop­ment process. In 2000, Graham et al⁵ compared different practice guideline appraisal instruments and identified 15, containing 8 to 14 questions/statements each. The authors grouped the state­ments into 10 important attributes to assess the methodological rigor for guideline development: validity, reliability/reproducibil­ity, clinical applicability, clinical flexibility, multidisciplinary pro­cess, clarity, schedule review, dissemination, implementation, and evaluation. Each instrument was independently examined. Only 2 were validated: the Cluzeau instrument⁶ received the best eval­uation, and constituted the basis of the Appraisal of Guidelines for Research and Evaluation in Europe (AGREE II) instrument,⁷ and the Shaneyfelt instrument⁸ was the other tool validated. More recently, Schünemann et al⁹ systematically compiled a com­prehensive checklist of items that guideline development tools could consider. This checklist included 18 topics to serve as a resource for guideline developers.

Appropriate Guideline Use

References targeting guideline workgroups provide an exhaustive list of what must be considered in the development process.^5-9 However, from the perspective of guideline users, decision mak­ers, and patients, a thoughtful appraisal of guidelines should be undertaken in order to apply this knowledge tool in appropriate circumstances. We propose asking the following questions: 1) Is there a need for a guideline? 2) Are the scope, analytical frame­work, and key questions relevant to my clinical reality? 3) Are the recommendations applicable in my clinical reality? 4) Is the process of recommendations development reliable and valid?

Is There a Need for a Guideline?

In order to evaluate if there is a need for a guideline, decision makers and patients should answer the following questions: What are the importance and burden of the disease? Is there a care gap based on evidence, and can outcomes be improved? Is there practice variation or clinical uncertainty?

In 2012, Atkins et al¹⁰ conducted a literature review to iden­tify the criteria for whether a topic should be considered for guideline development. While they found only a few studies that directly addressed this issue, they did identify a substantial con­sensus regarding the general factors that should be considered. The Institute of Medicine¹¹ outlined 6 criteria: prevalence, illness burden, potential to improve outcomes (care gap), management costs, cost reduction, and practice variation.

To document the prevalence and burden of a given illness, Atkins et al¹⁰ recommended the use of national data or review articles. The authors also suggested ad hoc stakeholders’ con­sultations to evaluate practitioner interest. To document the availability of evidence, Atkins et al¹⁰ recommended that existing reviews be reviewed and that preliminary literature searches be performed. Nonetheless, the question remains as to whether ob­servational studies, case series, animal studies, and case reports are sufficient to develop a clinical practice guideline. Random­ized controlled trials are not always possible. For example, in the EXTRIP (EXtracorporeal TReatments In Poisoning) guideline methodology,¹² the authors mention: “The guidelines are intend­ed to rely on evidence whenever possible. However, (...) extracor­poreal therapies have been recognized as the gold-standard treat­ment of several specific poisonings (...) elements of standard care in the management of poisonings and clinicians support its use overwhelmingly. In such instances a randomized trial would not be feasible or ethical.”

When the evidence is low or very low, the transparency of the consensus process is of primary importance, and the benefits of making recommendations (ie, decreased practice variation, facil­itation of future research, decreased costs) should outweigh the risk of harm (ie, recommending a harmful intervention).

Are the Scope, Analytical Framework, and Key Questions Relevant to My Clinical Reality?

After ensuring that there is a need for a guideline, decision mak­ers and patients should evaluate if the scope, analytical frame­work, and key questions are relevant to their clinical reality. They should answer the following questions: Is it the appropriate pop­ulation of patients and providers? Are all relevant options taken into account? Are all relevant key questions considered? Are pa­tient-centered health outcomes considered?

After ensuring that a guideline has been developed to answer a specific need (eg, to decrease burden of illness, decrease care gap, decrease practice variation), guideline users, decision mak­ers, and patients should evaluate the circumstances in which it applies. The scope of a guideline describes the populations of patients and providers, the type of services, and the sites for which the recommendations are intended.¹³ Some authors have raised concerns that guidelines may not properly consider the care of patients with comorbidities.^14,15 Therefore, investigators from the Improving Guidelines for Multimorbid Patients Study Group established consensus-based recommendations for each step of guideline development to address this issue. Guideline developers should consider how disease—disease, disease–treat­ment, and treatment–treatment interactions impact clinical man­agement and outcomes.

Are the Recommendations Applicable to My Clinical Reality?

After ensuring that a necessary guideline targets a scope and key questions relevant to their clinical reality, guideline users, deci­sion makers and patients should evaluate if the recommenda­tions are applicable, and answer the following questions: Are the recommendations based on evidence relevant to my reality? Do the recommendations take into account the resources I have? Do the recommendations consider the values and preferences of my patients and their relatives?

First, guideline users, decision makers, and patients should as­sess if they agree with the proposed evaluation of quantity and quality of evidence. The AGREE collaboration⁷ suggests that guideline workgroups use systematic methods to search for evi­dence related to each key question of the analytical framework, clearly describe the criteria for selecting the evidence, and pro­vide a detailed description of the strengths and limitations of the body of evidence. If a systematic review has already been pub­lished, the guideline developers should appraise its quality based on the Assessment of Multiple Systematic Reviews (AMSTAR).¹⁶ In circumstances where there are no available systematic reviews, guideline developers should perform a new synthesis of the ev­idence.

When a new systematic review is required, the American Heart Association Clinical Practice Guideline Methodology Group¹⁷ recommends incorporating the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) method­ology to evaluate the quality of evidence for each outcome across studies.¹⁸ The quality can be evaluated as “high” when there is confidence that the true effect lies close to that of the estimate effect (as often occurs for randomized trials), “moderate” when the true effect is likely to be close to the estimate of the effect but there is a possibility that it is substantially different, “low” when the true effect may be substantially different (as is often the case for observational studies), and “very low” when there is little confidence in the effect estimate.¹⁹

While some committees may consider not including a case se­ries in their synthesis of evidence, Chambers et al²⁰ emphasized that it can strengthen the credibility of a review for emerging interventions. They indicate the importance of case series as a useful source of evidence concerning safety, and they recognized the biases inherent in this study design, but reinforced the fact that non-randomized controlled trial evidence is often required to ensure clinical credibility of the review. Concerning the inclu­sion of animal studies in guideline development, Lamontagne et al²¹ recognized that animal research may inform clinical practice. When assessing literature reviews of therapeutic interventions for sepsis, they found that 27% of articles assumed that data from preclinical studies could apply to human patients. Some au­thors may not perceive the benefit of looking for articles with a weak level of evidence, but Guyatt et al²² mentioned: “Neverthe­less, reasons to consider a structured approach to formulating weak recommendations are: 1) for physicians who would appre­ciate guidance, providing that guidance; 2) specifying that a rec­ommendation is weak will reduce the likelihood that it will be in­appropriately considered as a measure of performance or quality of care; 3) signaling clinicians that patient values and preferences are likely to play a large role in these instances; and 4) helping in the justification for the research community of further research.”

Secondly, guideline users and decision makers may consider recommendations as not applicable if the required resources (hu­man or financial) are not available. Guyatt et al¹⁸ recommended that decisions first be made after considering the quality of evi­dence, the balance between desirable and undesirable outcomes, and values and preferences. The guideline development work­group should then consider whether any revision of direction or strength is necessary after considering resource use or costs.

Finally, a recommendation may not be applicable if it does not correspond to patients’ values and preferences. To ensure that values and preferences are considered, the Institute of Med­icine¹¹ recommends involving a former patient, patient advocate or patient organization representative in the guideline develop­ment process. Their participation may contribute to making the recommendations more acceptable to the patient population and, thus, making guideline implementation more likely.²³ However, including public views is often challenging due to a tendency to make hypothetical judgments that can conflict with the interests of patients and caregivers. Montori et al²⁴ stressed the fact that a committee development panel should at least avoid making strong recommendation when the best course of action heavily depends on the patient’s values and preferences.

In order to facilitate appropriate participation, the steering committee for patients and public involvement of the Guide­lines International Network²⁵ propose first identifying whether the goal is to consult patients to consider their values and pref­erences, and to directly solicit their participation or efficiently communicate recommendations, so they can provide their input for decision making. In addition, consideration of the values and preferences of other stakeholders—such as care deliverers, healthcare managers, governments, health insurers, employers or manufacturers—may also be relevant.⁶ This is particularly the case if they have a direct interest in the guideline, and particu­larly if their participation may impact the implementation of the recommendations.

Is the Process of Recommendations Development Reliable and Valid?

A necessary guideline containing applicable recommendations, and targeting a scope and key questions relevant to the clinical re­ality of guideline users, decision makers, and patients still should not be used if the development process is invalid or unreliable. Guideline credibility largely depends on the composition of the expert panel and whether they would be affected by potential conflicts of interest. The Canadian Medical Association² and the Institute of Medicine¹¹ suggest inclusion in the working group of a facilitator, representatives from professional groups that would be significantly affected by the guideline, clinicians, meth­odological experts, patient representatives, policy makers, and an administrative assistant. Funders should have no role in the guideline development, and chairs or co-chairs should not have any conflicts of interest.¹¹

Boyd et al²⁶ define a conflict of interest as, “when an individ­ual’s personal interests (eg, direct and indirect financial or intel­lectual) have the potential to compete with or influence behavior related to the individual’s professional interests or obligations (ie, evaluating the evidence and drafting recommendations for clini­cal practice guidelines).” Unfortunately, guidelines are not always as transparent as they should be. Choudhry et al²⁷ published the results of a cross-sectional survey of 192 authors of 44 guide­lines endorsed by North American and European societies be­tween 1991 and July 1999. On average, 81% of the guidelines’ authors had some form of interaction with drug manufacturers; 59% had relationships with companies that commercialized one of the drugs being considered for the guideline. Although only 7% of authors perceived that their interactions influenced the recommendations, 19% of their coauthors thought it did.

Consequently, Guyatt et al²⁸ proposed an approach to address conflicts of interest. First, they suggest that equal emphasis be placed on intellectual and financial conflicts, and explicit crite­ria should be provided for both. Second, they recommend that the methodologists in charge of each question or subject should not be conflicted. Finally, they emphasize that experts with im­portant financial or intellectual conflicts of interest can collect and interpret evidence, but not develop a recommendation for a specific question. Boyd et al²⁶ proposed a matrix for considering perception related to both commercial sponsorship and conflicts of interest. When there is both commercial sponsorship and con­flicted members, the risk is perceived as high and that member should be excluded from the process. When the same conflicted member is involved without commercial sponsorship, the risk is considered to be moderate. They suggest that all processes of guideline development be documented carefully to ensure that other committee members will balance the conflicted member’s views. When there are no conflicted members but commercial in­volvement is present, the risk is still moderate, but identification of alternative funding sources is recommended.

Lenzer et al²⁹ identified red flags for knowledge users that should raise suspicion of bias that could potentially be intro­duced as a result of commercial sponsorship or conflict of inter­est: the sponsor(s) is a professional society that receives substan­tial industry funding; the sponsor is a proprietary company or is undeclared or hidden; the committee chair(s) have any financial conflict; multiple panel members have any financial conflict; any suggestion of committee stacking that would preordain a recom­mendation regarding a controversial topic; no or limited involve­ment of an expert in methodology in the evaluation of evidence; no external review, and no inclusion of nonphysician experts, patient representative, or community stakeholders.

Table

Nonetheless, even with nonconflicted workgroup members, the consensus method must be very transparent and rigorous. The variety of different methods that can be used to formulate recommendations to integrate the evidence, costs, use of re­sources, values, and preferences for a given treatment are pre­sented in the .

When building recommendations based on a low level of evidence, Kunz et al³⁰ favored a formal consensus process that provides an opportunity for an active and equal involvement for all participants, notably because this strategy controls articulate members and reduces the power of strong individuals. A formal consensus also allows panel members to retract any firmly stated opinions. The objective of a formal consensus method is to iden­tify a central tendency among the panel and grade the level of agreement.³¹ A consensus definition should be specified a priori and reported.³²

Once an agreement is reached, each recommendation should provide a clear description of the potential benefits and harms; a summary of relevant available evidence (quantity and quality) and evidentiary gaps; an explanation of the part played by values, opinions, theory, and clinical experience; a rating of both the level of confidence and the strength of the recommendations; and a description of any difference in opinion regarding the recommendation.¹¹ An approach should be suggested to facili­tate guideline adherence¹⁴ and the results of the external review process should be made available⁷ so guideline users, decision makers, and patients can appropriately use the recommendations.

Conclusions

Given the potential guideline benefits and harms, it is of primary importance to ensure a rigorous development process as detailed by Schünemann et al.⁹ Moreover, guideline users, decision mak­ers, and patients should be aware of how to judge whether a recommendation is applicable to their specific circumstances in order to avoid an inappropriate use. They should evaluate if the guideline is needed, as well as if the scope, the analytical frame­work, and the key questions are relevant to their clinical reality; also if the recommendations are applicable, and if the process of recommendations development is reliable and valid. By em­powering those key stakeholders on how to use that important knowledge tool, guideline developers will be able to influence behaviors for the benefit of the population.

Acknowledgments

The author wishes to acknowledge Laurie J. Morrison and Na­dine Shehata for their scientific contribution, and Jennifer But­ters for language help.

Author Affiliation: Québec Poison Centre, Québec city, Canada; Université Laval, Québec city, Canada.

Funding Source: None.

Author Disclosures: Dr St-Onge reports no relationship or finan­cial interest with any entity that would pose a conflict of interest with the subject matter of this article.

Authorship Information: Concept and design; drafting of the man­uscript; critical revision of the manuscript for important intel­lectual content; administrative, technical, or logistic support; and literature review.

Address correspondence to: Maude St-Onge, MD, PhD, FRCPC, Medical Director of the Quebec Poison Centre, Assistant pro­fessor, Université Laval, 715 ave De Norvège, app 1, Québec, QC, Canada G1X3G6. Maude.st-onge.1@ulaval.ca.

REFERENCES

1. Graham ID, Logan J, Harrison MB, et al. Lost in knowledge translation: time for a map? J Contin Educ Health Prof. 2006;26(1):13-24.

2. Davis D, Goldman J, Palda VA. Handbook on clinical practice guidelines. Cana­dian Medical Association website. https://www.cma.ca/Assets/assets-library/doc­ument/en/clinical-resources/CPG%20handbook-e.pdf. Published 2007. Accessed November 2015.

3. Field MJ, Lohr KN (eds); Committee to Advise the Public Health Service on Clin­ical Practice Guidelines; Institute of Medicine. Clinical Practice Guidelines: Directions for a New Program. Washington, DC: National Academies Press; 1990.

4. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Potential benefits, lim­itations, and harms of clinical guidelines. BMJ. 1999; 318(7182):517-530.

5. Graham ID, Calder LA, Hébert PC, Carter AO, Tetroe JM. A comparison of clinical practice guideline appraisal instruments. Int J Technol Assess Health Care. 2000;16(4):1024-1038.

6. Cluzeau F, Littlejohns P, Grimshaw JM, Feder G, Moran SE. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care. 1999;11(1):21-28.

7. Chuang R, Heard KJ. Types of studies used to support treatment recommenda­tions in medical toxicology. J Med Toxicol. 2010;6(3):345-348.

8. Shaneyfelt T, Mayo-Smith M, Rothwangl J. Are guidelines following guidelines? the methodological quality of clinical practice guidelines in the peer-reviewed med­ical literature. JAMA. 1999;281(20):1900-1905.

9. Schünemann HJ, Wiercioch W, Etxeandia I, et al. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ. 2014;186:E123-E142.

10. Atkins D, Perez-Padilla R, MacNee W, Buist AS, Cruz AA; ATS/ERS Ad Hoc Committee on Integrating and Coordinating Efforts in COPD Guideline Develop­ment. Priority setting in guideline development: article 2 in integrating and coor­dinating efforts in COPD guideline development—an official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):225-228.

11. Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E. Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011: 391.

12. Lavergne V, Nolin TD, Hoffman RS, et al. The EXTRIP (EXtracorporeal TReatments In Poisoning workgroup: guideline methodology. Clin Toxicol (Phila). 2012;50(5):403-413.

13. Harris RP, Helfand M, Woolf SH, Lohr KN, Mulrow CD, Teutsch SM, Atkins D; Methods Work Group, Third US Preventive Services Task Force. Current methods of the US Preventive Services Task Force: a review of the process. Am J Prev Med. 2001;20(suppl 3):21-35.

14. Pronovost PJ. Enhancing physicians’ use of clinical guidelines. JAMA. 2013;310(23):2501-2502.

15. Uhlig K, Leff B, Kent D, et al. A framework for crafting clinical practice guide­lines that are relevant to the care and management of people with multimorbidity. J Gen Inter Med. 2014;29(4):670-679.

16. Shea BJ, Grimshaw JM, Wells GA, et al. Development of AMSTAR: a measure­ment tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol. 2007;7:10.

17. Jacobs AK, Kushner FG, Ettinger SM, et al. ACCF/AHA clinical practice guide­line methodology summit report: a report of the American College of Cardiology Foundation/American Heart Association Task Force on Practice Guidelines. Circu­lation. 2013;127(2):268-310.

18. Guyatt G, Oxman AD, Akl EA, et al HJ. GRADE guidelines: 1. introduc­tion—GRADE evidence profiles and summary of finding tables. J Clin Epidemiol. 2011;64(4):383-394.

19. Balshem H, Helfand M, Schünemann HJ, et al. GRADE guidelines: 3. rating the quality of evidence. J Clin Epidemiol. 2011;64(4):401-406.

20. Chambers D, Rodgers M, Woolacott N. Not only randomized controlled trials, but also case series should be considered in systematic reviews of rapidly develop­ing technologies. J Clin Epidemiol. 2009;62(12):1253-1260.

21. Lamontagne F, Briel M, Duffett M, et al. Systematic review of reviews includ­ing animal studies addressing therapeutic interventions for sepsis. Crit Care Med. 2010;38(12):2401-2408.

22. Guyatt G, Montori V, Schunemann H, et al. Letter reply to GRADE guidelines articles 14 and 15. J Clin Epidemiol. 2014;67(2):240.

23. Kelson M, Akl EA, Bastian H, et al. Integrating values and consumer involve­ment in guidelines with the patient at the center: article 8 in integrating and coor­dinating efforts in COPD guideline development. an official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):262-268.

24. Montori VM, Brito JP, Murad MH. The optimal practice of evidence-based medicine: incorporating patient preferences in practice guidelines. JAMA. 2013;310(23):2503-2504.

25. Knaapen L, Schaefer C, Boivin A, et al. G-I-N PUBLIC toolkit: patient and public involvement in guidelines. Guidelines International Network website. http:// www.g-i-n.net/document-store/working-groups-documents/g-i-n-public/toolkit/ toolkit-combined.pdf. Accessed November 2015.

26. Boyd EA, Akl EA, Baumann M; ATS/ERS Ad Hoc Committee on Integrating and Coordinating Efforts in COPD Guideline Development. Guideline funding and conflicts of interest: article 4 in integrating and coordinating efforts in COPD guideline development. an official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):234-242.

27. Choudhry NK, Stelfox HT, Detsky AS. Relationships between authors of clin­ical practice guidelines and the pharmaceutical industry. JAMA. 2002;287(5):612- 617.

28. Guyatt G, Akl EA, Hirsh J, et al. The vexing problem of guidelines and conflict of interest: a potential solution. Ann Intern Med. 2010;152(11):738-741.

29. Lenzer J, Hoffman JR, Furberg CD, Ioannidis JP; Guideline Panel Review Work­ing Group. Ensuring the integrity of clinical practice guidelines: a tool for protect­ing patients. BMJ. 2013;347:f5535.

30. Kunz R, Fretheim A, Cluzeau F, et al; ATS/ERS Ad Hoc Committee on In­tegrating and Coordinating Efforts in COPD Guideline Development. Guideline group composition and group processes: article 3 in integrating and coordinating efforts in COPD guideline development. an official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):229-233.

31. Nair R, Aggarwal R, Khanna D. Methods of formal consensus in classification/ diagnostic criteria and guideline development. Semin Arthritis Rheum. 2011;41(2):95- 105.

32. Diamond IR, Grant RC, Feldman BM, et al. Defining consensus: a systematic review recommends methodologic criteria for reporting Delphi studies. J Clin Epi. 2014;67(4):401-409.

Download PDF: Using Guidelines: When Is It Appropriate?