Françoise Cluzeau explains how the Health Care Evaluation Unit at St Geroge's Hospital Medical School appraise national clinical guidelines

Clinical guidelines have become a central feature of modern medicine. Their influence pervades all aspects of healthcare, from clinical decisions about management of individual patients to policy making.1,2 Their potential benefits are enormous.

If appropriately developed and implemented, guidelines can improve health outcomes by promoting interventions of proven efficacy.3 They can offer explicit and consistent advice to clinicians about particular treatments.4 Many believe that they can also be used as levers to improve efficiency and contain spiralling healthcare costs.

On 1 April 1999, the NHS launched the National Institute for Clinical Excellence (NICE). One of its key functions is to develop and disseminate clinical guidelines for national use.5

Despite this enthusiasm there is a growing feeling that many barriers need to be overcome before guidelines can achieve their outcome.4,6 One area of general concern is the wide variation in the quality of guidelines.7

The proliferation of guidelines has led to the duplication of guidelines on the same topic, and some contain inconsistent or even conflicting recommendations which confuse and frustrate clinicians.7,8 Unfortunately, the reasons for such discrepancies are often difficult to establish because information about how the guidelines have been developed is not sufficiently detailed.

Since clinical guidelines are designed to influence practice, their users need to be reassured that they are recommending effective treatment.9 Guidelines should therefore be assessed before they are recommended for widespread use.10

Assessment of clinical guidelines is complex and needs to be under- taken at several stages. An initial assessment requires that the guideline developers have minimised the potential biases in creating them.11

In 1995 the Health Care Evaluation Unit at St George's Hospital Medical School, in collaboration with the Health Services Research Unit from Aberdeen University and Queen Mary and Westfield College, embarked on the development of an instrument to appraise the methodological quality of a wide range of clinical guidelines, especially with regard to their wider application in the NHS.12

The instrument is presented as a checklist of 37 questions that address key aspects of guideline development, which could be mapped to the Institute of Medicine) attributes of 'good' guidelines (Table 1, below).13

Table 1: Appraisal instrument – selected key criteria

Rigour of development:

Is the agency responsible for the development of the guidelines clearly identified?

Did the development group contain representatives of all key disciplines?
Are the sources of information used to select the evidence adequate?
Are the methods for assessing and rating the evidence adequate?

Are the methods for formulating the recommendations satisfactory?

Is there an explicit link between the major recommendations and the level of supporting evidence?

Is there a date for reviewing or updating the guidelines?

Dimension two. Context and content:

Are the reasons for developing the guidelines and their objectives clearly stated?

Is there a satisfactory description of the patients to be covered by the guidelines?
Is there a description of the circumstances in which exceptions might be made in using the guidelines?
Is there a statement of how the patient's preferences should be taken into account in applying the guidelines?

Are the recommendations clearly presented?

Are the recommendations supported by the estimated benefits, harms and costs of the intervention?

Dimension three. Application of guidelines:

Does the guideline document suggest possible methods for dissemination and implementation?

Does the guideline document identify key elements which need to be considered by local guideline groups?

Does the guideline document identify clear standards or targets?

Does the guideline document define measurable outcomes that can be monitored?

We validated the instrument on 60 national and local UK guidelines using 120 appraisers, in a study funded by the National Research and Development Programme. Details of the study have been published elsewhere.14

This work coincided with the publication by the NHS Executive of its policy document on the development, appraisal and application of clinical guidelines.15 The document explicitly supported the development of high-quality guidelines. This marked the beginning of a national programme for the development and endorsement of national guidelines.

As part of this programme an independent appraisal service was established,16 and the Health Care Evaluation Unit was commissioned to run this appraisal service for a period of 2 years until the end of July 1999. The function of this service is to provide advice to the NHS Executive about the quality of the guidelines which it funds through its national programme.

The appraisal is essentially a structured peer review based on the same methodology as the one developed in the research study. Each guideline is appraised by at least six appraisers. Their professional affiliation varies depending on the subject covered by the guidelines, but the appraisers usually comprise a hospital consultant, a GP, a nurse, a public health consultant, a researcher/health economist, a manager, and a pharmacist.

The appraisers must fulfill at least three criteria:

  • They must be independent of the guidelines
  • They must have expert knowledge or clinical experience of the topic covered by the guidelines
  • They must be based in the UK.

The appraisals are carried out independently. A full pack, containing a letter with instructions, the guidelines, additional background literature, the appraisal instrument and its user guide, is sent to the appraisers. They are given 3 weeks to complete the task.

Áe encourage the appraisers to qualify their responses on the space provided in the checklist. These comments provide valuable background information on how they reached a decision and why they may disagree. It also allows them to express their views on specific issues.

Information from the appraisals are synthesised in a report, comprising four sections:

  • An Executive Summary for the NHS Executive, the NGG or other decision-makers. A pr³cis of the background and development of the guideline is also provided.
  • A textual summary of the appraisers' responses from the checklist.
  • Distribution of the appraisers' responses for each criterion.
  • A transcription of the general comments made by appraisers about the guidelines, or the appraisal methodology.

The appraisal reports are completed within 10–12 weeks of receiving the guidelines. The reports are sent to the NHS Executive, to the guidelines' authors, and to the appraisers for feedback.

Up to now, 20 guidelines have been appraised through the service (Table 2). The majority were produced by the Royal Colleges and five had been developed by the North of England Evidence Based Guideline Development Project.

Table 2: Number of guidelines achieving the criteria

Selected criteria (question number in instrument)
Appraised guidelines (n=20) (%)
Agency responsible clearly identified (01)
20 (100)
Development group representative (05)
2 (10)
Sources of information adequate (07)
14 (70)
Methods for assessing the evidence adequate (09)
10 (60)
Methods for formulating the recommendations satisfactory (11)
3 (15)
Link between evidence & recommendations (13)
15 (75)
Objectives clearly stated (21)
15 (75)
Recommendations clearly presented (28)
18 (90)
Estimates of costs and expenditure provided (31)
3 (15)
Dissemination methods suggested (33)
3 (15)
Clear standards or targets identified (35)
1 (0)

The methodology used to select the reviewers is designed to minimise the potential biases and conflict of interest that individuals may have. With the exception of one or two cases, we have found that the appraisals are fair and balanced. Some reviewers produce more detailed appraisal reports than others, but it is clearly impossible to control for such differences as they may be due to many factors (such as time constraints or personal characteristics).

We have found that recent guidelines are better documented than those appraised 2 years ago. In particular, there has been a substantial improvement in the way that the search strategy and the methods for assessing the evidence are reported. Also, at least two of the guidelines contained estimates of costs and expenditure.

We believe that this improvement is due to two factors:

  • Guideline producers have a better knowledge of the principles of 'good guideline development'. The NHS policy document, along with the several publications on guideline methodology, have provided clear guidance that has been perceived to be useful. Also, there is evidence that the appraisal instrument itself is becoming more widely used as an aide memoire to develop guidelines.
  • The prospect of a formal appraisal acts as an incentive for guideline authors to provide explicit and detailed documentation.

Despite this improvement, some aspects of guideline development remain opaque. For example, the methods used to formulate the recommendations are often not explicit or not adequately documented. This area needs attention as most recommendations are often not based on strong evidence, and therefore rely on some consultation or consensus at the final stage.10

Guideline developers also need to pay closer attention to the issue of costs.

Furthermore, there appeared to be no improvement in the strategy for disseminating, implementing and monitoring the guidelines. Many believe that these issues are best addressed at a local level.17

It is important that users of guidelines should have ready access to information on the quality of guidelines once they have been assessed. Unfortunately, at present there is no national database of appraised guidelines in England.

Owing to the increasing demand for information, there are currently a number of private and commercial initiatives to develop software that will make guidelines available to the public domain through the internet. The quality of the majority of these guidelines will probably be unknown. Yet clinicians could be challenged in their decisions by patients who have access to this information.

It is therefore desirable to have an official NHS-recognised database/ clearinghouse that will contain the guidelines and their appraisal, to promote widespread access. This could be based on a similar model as the one recently established for the guidelines clearinghouse in the US.18

The National Centre for Clinical Audit in the UK has been investigating the setting up of an electronic database of UK guidelines, and there will be obvious advantages in linking with the National Clearinghouse.

The assessment of guidelines is an integral part of a national guidelines programme. In a recent discussion paper, the NHS made it clear that guidelines will be subjected to a formal appraisal before they are recommended for use in practice.5

Ensuring that guidelines are methodologically sound may not be the only way to ensure that they will be effective, but it is a necessary step towards that process.

Details of the instrument and appraisal methodology are available on the Health Care Evaluation Unit website: http://www.sghms.ac.uk/phs/hceu/. A list of the appraised guidelines and summaries of the appraisal reports are also available on this website.

  1. Kleijnen J, Bonsel G. Guidelines and quality of clinical services in the new NHS. Br Med J 1998; 316: 299-300.
  2. West E, Newton J. Clinical guidelines: an ambitious national strategy. Br Med J 1997; 315: 324.
  3. Grimshaw JM, Russell IT. Achieving health gain through clinical guidelines II: Ensuring guidelines change medical practice. Q Health Care 1994; 3: 45-52
  4. Woolf SH, Grol R, Hutchinson A, Eccles M, Grimshaw J. Potential benefits, limitations, and harms of clinical guidelines. Br Med J 1999; 318: 527-30.
  5. NHS Executive. Faster Access to Modern Treatment: how NICE appraisal will work. A discussion paper. Department of Health, 1999.
  6. McKee M, Clarke A. Guidelines, enthusiasms, uncertainty, and the limits to purchasing. Br Med J 1995; 310: 101-4.
  7. Thomson R, McElroy H, Sudlow M. Guidelines on anticoagulant treatment in atrial fibrillation in Great Britain: variation in content and implications for treatment. Br Med J 1998; 316: 509-13.
  8. Hutchinson M. More advice on hypertension. Doctor, 11 February 1999, p 17.
  9. Weingarten S. Practice guidelines and prediction rules should be subject to careful clinical testing. JAMA 1997; 277: 1977-8
  10. Hayward RS, Wilson MC, Tunis SR, Bass EB, Guyatt G. Users' guides to the medical literature. VIII. How to use clinical practice guidelines. A. Are the recommendations valid? The Evidence-Based Medicine Working Group. JAMA 1995; 274: 570-4.
  11. Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Clinical guidelines: Developing guidelines. Br Med J 1999; 318: 593-6.
  12. Cluzeau F, Littlejohns P, Grimshaw J, Feder G. Appraisal Instrument for Clinical Guidelines Version 1. London: St George's Hospital Medical School, 1997.
  13. Field MJ, Lohr KN (eds) Guidelines for Clinical Practice. From Development to Use. Washington DC: National Academy Press, 1992.
  14. Cluzeau F, Littlejohns P, Grimshaw J, Feder G, Moran S. Development and application of a generic methodology to assess the quality of clinical guidelines. Int J Qual Health Care 1999; 11: 21-7.
  15. NHS Executive. Clinical Guidelines: Using clinical guidelines to improve patient care within the NHS. Leeds: NHS Executive, 1996.
  16. Department of Health. Clinical guidelines and independent appraisal. In: Chief Medical Officer's Update. Department of Health, 1997; p. 8.
  17. Wolfe CDA, Stojcevic N, Rudd AG, Warburton F, Beech R. The uptake and costs of guidelines for stroke in a district of southern England. J Epidemiol Community Health 1997; 51: 520-5.
  18. Atkins D, Kamerow D, Eisenberg JM. Evidence-based Medicine at the Agency for Health Care Policy and Research. 1998; 98-RO44.

Guidelines in Practice, June 1999, Volume 2
© 1999 MGP Ltd
further information | subscribe