Guidelines for the Development of Practice Guidelines

March 1998

There is growing use of practice guidelines as a means of reducing inappropriate care, controlling geographic variations in practice patterns, and making more effective use of health care resources. Many specialty societies and other professional organizations, including the American College of Rheumatology, have developed guidelines. The methods by which these guidelines have been developed, as well as the purpose, dissemination, audience and ultimate use of each one, have varied considerably. The question of whether or not the American College of Rheumatology should develop guidelines is not the purpose of this document. This is an attempt to outline how the ACR might go about ensuring consistency in the production of future guidelines. It is not an exhaustive review of guideline production, but rather a review of the development alternatives.

There are basically 4 major approaches to guideline development: informal consensus, formal consensus, evidence-based, and explicit development. Informal consensus is the product of a movement where guidelines are created almost entirely on the basis of expert opinion. These guidelines typically provide only recommendations and little background on the process by which they were developed. Evidence is often cited in the discussion, but there is usually very little information provided on the methodology used in developing the guideline that would assure readers that the science was reviewed without bias and that the recommendations were actually influenced by the evidence.

While informal consensus remains the most common approach to developing guidelines there are many problems with it. The appeal is that it is relatively easy, fast and free of complex analytic methods. The problems are that the guidelines are often of poor quality for several reasons. The first is that there are fundamental limitations to the validity of expert opinion as a basis for defining what is appropriate. The fact that a group of experts thinks that a practice is beneficial does not ensure that it is actually beneficial. Second, the lack of explicit methodology raises questions about how consensus is achieved. Guidelines that are produced in group meetings without systematic procedures are easily influenced by group dynamics, dominant personalities, and organizational and specialty politics. Third, the absence of documented methods makes it difficult for readers to judge whether the guidelines were influenced by scientific evidence or whether the evidence was overlooked due to panel biases. For example, if a specialty society recommends a procedure that generates income for its members, questions about potential conflicts of interest will be raised if the methods and rationale are not well documented.

The formal consensus methodology was introduced in the mid–1970s. This is best exemplified by the NIH formal consensus approach in which an expert panel reaches consensus on recommendations in a structured 2 ½ day conference. The NIH approach has been repeated for more than 15 years on more than 80 topics, but although it provides greater structure to the analytic process than does the informal consensus process, the absence of explicit criteria and the requirement that recommendations be produced quickly within a single meeting have been greatly criticized.

The RAND Corporation developed a more formal approach to consensus development in the 1980s. An expert panel is provided with background articles that review existing scientific evidence for a procedure, as well as a list of hundreds to thousands of potential indications for performing the procedure. Then they use a 2–step Delphi technique. Before the first meeting, panel members are asked to assess the appropriateness of the procedure for each indication, using a 9-point scale. Then, when the panel meets, members consider the group scores with their own judgments and discuss reasons for disagreement. Panel members then repeat the scoring process, revising their scores on the basis of the group discussion. The final list of appropriateness ratings reflects the extent of agreement of the panelists about the appropriateness of the procedure for each indication.

While the RAND approach has proved popular, it has some limitations. The product of the approach – a list of appropriateness scores – is often difficult for clinicians to apply in practice. The approach also retains the fundamental limitations discussed above in terms of the use of opinions as the basis for defining appropriateness.

Evidence based guidelines are illustrated in the Clinical Efficacy Assessment Project of the American College of Physicians. The US Preventative Services Task Force has also introduced explicit criteria and systematic methods for reviewing evidence of clinical effectiveness. The evidence-based approach has rightly been credited with enhancing the scientific rigor of guideline development, but the problem with this approach is that it is often unable to produce recommendations in the absence of credible evidence. This results in neutral recommendations that often do not help the clinician with decision making. In fact, if strict adherence to an evidence-based approach was kept, then the vast majority of medicine would be excluded from consideration because of the lack of clinical studies to back up much of medical practice. The AHCPR had originally taken an approach that combined a detailed, evidence-based approach with a process that accommodates expert opinion. This approach met with some success until recent political problems derailed it. The process included a formal assessment of scientific evidence, panel meetings and open forum sessions that sought input from a broad constituency on relevant issues.

Eddy has promoted the use of more explicit methods of guideline development. In his approach, guideline developers specify the benefits, harms and costs of potential interventions and derive explicit estimates of the probability of each outcome. Whenever possible, scientific evidence and formal analytic methods are used to generate the estimates. The estimates are also generated by the expert opinion, but the source of the estimate is documented. The assumptions are then tabulated in a balance sheet which allows readers of the guideline to review the potential benefits, harms and costs of each choice. The process of estimating outcomes is separated from the process of making judgments about the desirability of the outcomes. The latter involves patient preferences and is made by individuals. The main criticism of this approach is that it is complex and very time consuming.

There are basically 4 major steps to successfully completing a practice guideline: introductory decisions, assessments of clinical appropriateness, assessment of public policy issues, and guideline document development and evaluation.

Introductory Decisions:

There are 3 steps:

1. Selection of a Topic – These can be either diseases, complaints or procedures. They can focus on prevention, diagnosis, treatment or rehabilitation.
2. Selection of Panel Members – There are not set ways to select a panel. They should generally include physicians, but they can also include other health care professionals such as nurses, and methodologists such as health economists. Sometimes it is important to include patients and consumer representatives.
3. Clarification of Purpose – The definition of the topic is, in many ways, the most important decision to make. This includes specification of the target condition, the type of patients and clinical presentations for which the guidelines are intended, and the interventions to be considered. The panel needs to also consider the setting for which the guidelines are intended, including types of providers and practice settings.

There are 5 steps:

1. Assessment of Clinical Benefits and Harms – This should be the principal analytic task in developing guidelines. The two principal sources of information are scientific evidence and expert opinion.
2. Assessment of Scientific Evidence – The comprehensiveness of literature reviews varies in guideline development. Panels can limit evidence to published studies or can include unpublished evidence. Published evidence can be limited to clinical trials or can include other types of studies. Evidence review involves 3 steps:
1. Retrieval – This includes literature searches, etc.
2. Evaluation of Individual Studies – This involves combining evidence from different studies to reach conclusions about the overall weight of the evidence of a particular topic. This can involve informal as well as formal analytic approaches.
3. Synthesis of the Evidence – This involves combining evidence from different studies to reach conclusions about the overall weight of the evidence of a particular topic. This can involve informal as well as formal analytical approaches.
3. Assessment of Expert Opinion – The opinions of panel members can be assessed through informal or formal techniques to supplement information from the literature. In either case, the opinions and rationale of the experts needs to be documented explicitly so that readers are aware of which recommendations are based on evidence and which are based on opinion.
4. Summary of Benefits and Harms – This can be informal. It can use a balance sheet approach or be part of a formal decision analysis.
5. Determinations of Appropriateness – Information about benefits and harms is used to determine which practices are appropriate, which are inappropriate and which are in a gray zone. Drafting the language in the gray zone is one of the greatest challenges in writing practice guidelines. Cookbook medicine can be avoided by describing explicitly where there is uncertainty, by issuing recommendations that are broad and by allowing for legitimate differences in opinion in individual cases. This is an area where patient preferences should also be taken into account.

The above considerations focus on clinical benefits and harms and help to define the ideal. Guidelines that are intended for large groups, however, are public policies that can have a large impact on society. Thus, developers of guidelines need to consider the issues of resource limitations and feasibility. Resource limitation issues can be dealt with informally or through formal cost effectiveness analysis. The feasibility issues concern being able to implement recommendations in real world environments where patients and practitioners interact.

There are 4 issues that must be considered:

1. Drafting of Guideline Document – Guidelines should provide clear information about what they do: the proper method of performing procedures and administering treatments. They also need to review the methodology by which the guidelines were developed.
2. Peer Review and Pretesting – Draft guidelines should be peer–reviewed to ensure scientific and clinical validity. Sometimes it is advantageous to perform pretesting by asking a small group of practitioners to use the guidelines for a brief period and then collect their suggestions on ways to improve the document.
3. Recommendations for Dissemination, Evaluation, and Updating – A plan should be developed for disseminating the guidelines to the target audience. Publication will be considered separately below.
4. Recommendations for Research – The process of developing guidelines often uncovers important gaps in scientific evidence. The clarification of research needs is often the most important outcomes of guideline development.

A key issue in the development of practice guidelines is determining the most appropriate way of disseminating them. This always involves deciding upon the best way to reach the desired audience, and a key issue is that of publication – specifically where, and in what format, such publication should occur. One needs to consider the originality of the document, its scientific and research value, the nature and desires of the organization or group commissioning the guidelines, and any commercial funds used in the guideline preparation that might preclude publication in certain journals. Whenever possible, such publication should be considered in advance, especially when a guideline is being developed as a collaborative effort between two groups, each of which may have different ideas about the location or type of publication. However, even though publication considerations may be important to the individuals preparing a guideline (or the organization as a whole), the more important goal in guideline preparation is to develop a product that meets its original objectives, has clinical value, and will be used by those for whom it was intended.

Guidelines, once developed, must be reviewed periodically to ensure that they remain current. Each guideline should be evaluated each year to determine if changes are warrented by new developments in the field. In many cases, the guideline may not require revision.

Each year, the Committee on Rheumatologic Care will survey the lead authors of each guideline to determine whether the guideline should be revised. In the event that a lead author is not willing or is no longer available to review the guideline, CORC will designate a reviewer.

If the lead author determines that the guideline is still reflective of the best evidence and consensus, then there will be no revision of that guideline.

If the lead author determines that revision is necessary, CORC will ask the lead author to lead a revision task force of two or three people. CORC will consult with the lead author in choosing the revision task force. If the lead author chooses not to participate, CORC will appoint a replacement. The task force will be given a reasonable timeline for completing the revision. CORC will then review the revision and send it to the Board for final action. Once revisions are approved by the Board, they will be submitted to Arthritis & Rheumatism as letters to the editor. In the even that the guidelines change so substantially that a letter to the editor is not appropriate, an entire new manuscript will be submitted to the journal for publication.

Approved by the Board of Directors July 11, 1998