Our editorial board answer your questions on the meaning of 'evidence' and the importance of RCTs

Q Guidelines and reviews talk about 'lack of evidence' for certain treatments or forms of management. This seems to mean different things, from many good studies, giving no evidence of efficacy, to no studies covering the modality. How do we best analyse the meaning of such comments, which can carry so much weight in a guideline?

A As you say, the term 'evidence' can mean different things. On one dimension, it means anything from 'randomised controlled trials (RCTs)' to 'any material with numbers in it, including surveys of dubious value' to 'any well-intentioned enquiry'.

It is often used as a shorthand for 'quantitative', but qualitative evidence probably has an important place. Qualitative data can be analysed using well-developed social science techniques, but are never subject to the precision (some would say spurious precision) of quantitative data. Evidence can mean 'raw data' or 'raw data and interpretation'.

The more you think about it, the less definitive most 'evidence' is most of the time. The important thing is that any guideline should set out what the evidence is and where it came from, assess the strength of the studies on which the evidence is based, and assess the strength of the conclusions based on the evidence. Then at least the recommendations are clear for all to see and are open to challenge.

I find the outline shown in Table 1 (below), based on standard schemes, helpful.

Table 1: Levels of evidence
Category of evidence
Ia

Evidence from meta-analysis of randomised control trials

Ib

Evidence from at least one randomised controlled trial

IIa

Evidence from at least one controlled study without randomisation

IIb

Evidence from at least one other type of quasi-experimental study

III

Evidence from non-experimental descriptive studies, such as comparative studies, correlation studies and case-control studies

IV Evidence from expert committee reports or opinions or clinical experience of respected authorities, or both
Strength of recommendations
A Directly based on category I evidence
B Directly based on category II evidence or extrapolated recommendation from category I evidence
C Directly based on category III evidence, or extrapolated recommendation from category II evidence
D

Directly based on category IV evidence, or extrapolated recommendation from category III evidence

Adapted from Shekelle et al1
  1. Shekelle PG, Woolf SH, Eccles M, Grimshaw J. Developing guidelines. Br Med J 1999; 318; 593-6.

Q It seems that more and more guidelines are being produced, where the level of evidence behind the recommendations is moderate or low, i.e. not backed by significant numbers of high quality, randomised controlled trials (RCTs). Would it be better to ignore such guidelines until the necessary research has been done, rather than making widespread changes that affect practice and expenditure and are based on less definite evidence?

A In addition to what counts as evidence (see previous question & answer), there is often an assumption that 'evidence' means 'data' and 'conclusions', and that if there is evidence this is shorthand for 'clear evidence for or against the proposition'. And as well as the strength of the results in terms of clinical conclusions, there are the cost-effectiveness conclusions. So there are various combinations (see Table 2, below).

Table 2: Combinations of evidence and conclusions
Clinical effectiveness
Evidence for clinical effectiveness Yes / equivocal / no
Conclusions for clinical effectiveness Strongly positive / weakly positive / equivocal / weakly negative / strongly negative
Cost-effectiveness
Evidence for cost-effectiveness Yes / equivocal / no
Conclusions for cost-effectiveness Strongly positive / weakly positive / equivocal / weakly negative / strongly negative

Deciding what to do is straightforward where there is evidence based on a sound study design that leads to a strongly positive or negative conclusion about clinical- and cost-effectiveness.

However, as Table 2 shows, the information available may not be so clear. For some situations, evidence may never be forthcoming for ethical or practical reasons; in others, evidence may be a long time coming because research on more pressing problems takes priority.

In these situations, or where there is only evidence based on poor studies, and/or the conclusions are not clear-cut, the best advice is to take decisions that use what information there is, but to keep an open mind about what future studies may reveal and to encourage research that will provide the answers.

Evidence or not, patients still need care. Where best clinical care is not clear on the basis of evidence, we should use other techniques – such as consensus groups – to come up with a subjective but systematic view to guide practice until better information is available.

Q Bearing in mind the proliferation of guidelines and the clamour for more from all quarters, what is the optimal level at which guidelines should be developed and thus owned? Should we accept them direct from NICE, health authority, PCG/T, or should we concentrate on the few that we can develop in our practices?

A There probably isn't a level that is optimal for all guidelines. The answer depends on:

  • What aspect(s) of treatment and/or service organisation a specific proposed guideline covers; and
  • The availability of time and skills to do the necessary work.

In general, it is probably helpful to see guidelines as consisting of two parts:

  • The relevant evidence on the subject
  • Interpretation of the evidence, i.e. what it means in practical terms.

These practical conclusions fall into two groups: general (e.g. patients with blood pressure over a defined threshold should receive antihypertensive treatment) and particular (e.g. in your area, the formulary antihypertensive drugs are X, Y and Z; patients in certain groups should have the following investigations carried out by the GP before referral to a specialist; the specialist who deals with these types of cases is Ms A and her contact number is XXX).

The relative importance of these three components varies between subjects. It is probably best not to carry out general assessments, either of the literature at local level or the general practical recommendations that the evidence suggests. It is inevitable, however, that the final translation into working recommendations has to be carried out locally; it cannot be done satisfactorily at national level.

My advice is to use your energies locally to ensure that guidelines from elsewhere are understood and owned by you and your colleagues and that the recommendations make sense in terms of the available facilities and other opportunities and constraints.

Guidelines in Practice, March 2000, Volume 3
© 2000 MGP Ltd
further information | subscribe