Dr Mark Charny suggests a checklist for assessing whether a guideline is relevant and sound, in the fifth article in this series
When you have received a guideline from elsewhere, you should consider whether it is relevant to your circumstances. In particular, does it:
- Deal with an issue that is a local priority
- Deal with the sort of patients you see
- Describe a treatment context similar to the one in which you work.
If a guideline fails on any of these criteria, it is probably best to turn your attention elsewhere.
If it passes these tests, you need to ask other more detailed questions to help you decide whether the guideline is sound as well as apparently relevant. This will help to ensure that you do not spend time and effort implementing a guideline that has been poorly developed. Key questions are as follows.
1. Were an appropriate range of individuals involved in the development of the guideline?
This should reflect the professions concerned with the care for the condition described by the guideline. A guideline on stroke rehabilitation developed by neurologists alone, for example, is not likely to capture all of the aspects relevant to general practice, or care based in the community.
2. Were potential conflicts of interest taken into account? Do any of those who developed the guideline have any axe to grind?
An obvious conflict of interest to look out for is sponsorship of some or all of the individuals by a pharmaceutical company that makes a medicine included in the guideline, or a guideline based on a sponsored 'symposium'.
Less obvious is the inclusion of an individual who has pioneered a new technique and may be arguing more strongly for this than current evidence warrants.
These personal biases do not necessarily invalidate the guideline's recommendations, but if they exist you should treat the guideline with greater caution.
3. Was the guideline developed and endorsed by relevant groups or organisations?
While you should try to find out enough to make up your own mind about the extent to which you and your colleagues should rely on a guideline, you are unlikely to have the time or skills to satisfy yourself on every detail. The endorsement of relevant professional organisations will help to reassure you that what is being proposed is sound.
Once again, you should look for support from groups representing the professions who provide the care described in the guideline.
4. Were adequate steps taken to ensure that the search for evidence was comprehensive?
For example, if a guideline on treatment for snoring relied solely on reports published by a private snoring treatment clinic, we might treat the recommendations with a good pinch of salt.
It is difficult to know whether a search has been truly comprehensive. A good guideline will describe the way in which the evidence base was assembled. An outline is shown in Table 1 (below). A comprehensive search is an important protection against unconscious as well as conscious bias.
Table 1: Methods of assembling evidence
Cochrane controlled trials register
Centre for Reviews and Dissemination
National Institute for Clinical Excellence
National Guidelines Clearing House
National Centre for Clinical Audit
Oxford Guidelines Project
|National policy statements||
National Service Frameworks
5. Was an adequate method used to assess the quality of the scientific evidence?
Details of methods can be found, for example, in publications from the NHS Centre for Reviews and Dissemination or the Agency for Health Care Policy & Research.1 This question is really about the extent to which the evidence is susceptible to bias. An outline scheme is shown in Table 2.
|Table 2: Outline scheme of a method of assessing the quality of scientific evidence*|
|Category of evidence|
Evidence from meta-analysis of randomised control trials
Evidence from at least one randomised controlled trial
Evidence from at least one controlled study without randomisation
Evidence from at least one other type of quasi-experimental study
Evidence from non-experimental descriptive studies, such as comparative studies, correlation studies and case-control studies
|IV||Evidence from expert committee reports or opinions or clinical experience of respected authorities, or both|
|Strength of recommendations|
|A||Directly based on category I evidence|
|B||Directly based on category II evidence or extrapolated recommendation from category I evidence|
|C||Directly based on category III evidence, or extrapolated recommendation from category II evidence|
Directly based on category IV evidence, or extrapolated recommendation from category III evidence
|* Adapted from Chekelle et al2|
6. Are the guideline recommendations based on an adequate strength of evidence?
Given that the evidence is likely to be reasonably unbiased, how strong are the conclusions? An outline scheme is shown in Table 3.
Table 3: Examples of ambiguous and unambiguous statements from recommendations on management
|Ambiguous statement||Unambiguous statement|
|Elevated cholesterol||Serum cholesterol >6.5mmol/l|
|Steroids daily||Oral prednisolone 5mg tds|
Discharge within 12 hours postoperatively
Although the categorisation of evidence is hierarchical, and one should always use the highest level of evidence available, evidence is often lacking or ambiguous: we may need to depend on evidence as low as category IV. However, there will be much more room for a personal view about following the recommendation based on category IV evidence than on category I evidence.
7. Are there precise descriptions of the patients to whom the guideline is meant to apply?
'The elderly' is imprecise; 'all of those admitted under the care of a geriatrician' is less unclear; 'men aged 85+' is very precise. Precision is important because you need to know how closely your patients match those described in the guideline or to which of your patients the recommendations in the guideline are intended to apply.
8. Are the recommendations on management described in unambiguous terms? Similarly, if you decide to follow the guideline, do you know exactly what to do?
9. Does the guideline consider potential costs, risks, and benefits resulting from its use?
If a guideline on community care for people with severe psychotic illness is likely to be costly, high risk, and of dubious benefit to the patients concerned, you will think twice about using it.
Assessing a guideline with the degree of rigour expected, for example, from the Centre for Reviews and Dissemination or NICE, involves technical skills which most clinicians will not wish to acquire. However, completing a simple checklist using informed common sense will help to separate out self-serving guidelines. Any good guideline should make it easy to answer these questions.
An appraisal instrument for clinical guidelines (see Assessing the quality of national guidelines) is available from the NHS Appraisal Centre for Clinical Guidelines at St George's Hospital (http://www.sghms.ac.uk/depts/phs/hceu/clinguid.htm).
- Charny M. How to find out about existing guidelines. Guidelines in Practice 1999; 2 (May): 58-61.
- Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Developing guidelines. Br Med J 1999; 318: 593-6.