In the first of two articles on audit, Dr Mark Charny explains how applying a few simple principles can mean the difference between time well spent and time wasted


Clinical audit is – or should be – a straightforward practical activity. When it is conducted properly, it is an effective way to improve patient care and a rewarding means of professional development.

Unfortunately, many people ignore the basics and as a result waste their own – and their colleagues’ – time. Many audits that achieve nothing useful are also over-complicated. A successful audit is an encouragement to further efforts; an unsuccessful audit causes disillusionment and cynicism, neither of which is helpful to the participants or their patients.

The audit cycle

Figure 1 (below) shows the audit cycle. Three features are particularly worth noting:

  • The standards against which practice is measured are introduced into the audit, not generated within it. This important point is often overlooked. Many audits observe what is happening and from that generate standards. This is circular reasoning, and often generates the conclusion that we are meeting the standards – because they are defined as the care we have observed ourselves giving!
  • All the stages of the cycle are essential. Missing a stage eliminates any chance of an audit producing worthwhile results. Many audits are ‘action-lite’: someone is delegated to collect the audit data and the more senior participants do nothing with the findings other than simply to note them.
  • The figure should really be viewed in three dimensions, as an audit spiral. Quality improvement proceeds through small incremental gains rather than step changes in practice. There are occasions when a single audit cycle is sufficient, but more typically the first audit cycle reveals deficiencies in care, these are examined and action taken to rectify them. Standards are then raised, and a further audit examines whether care has improved against these new standards. This process may be repeated several times, depending on the gain anticipated from a re-audit. Re-audit does not mean going round and round in circles.
FIgure1: The audit cycle

Before you start

Objectives

Before you embark on an audit, you need to decide what you want to achieve, for example improve outcomes, control costs, identify unmet need, achieve greater efficiency, ensure your patients are better informed or avoid complaints. An audit is a means to an end, not an end in itself. No-one would set out on a journey without having a destination in mind; similarly, an audit without a clear objective will go nowhere.

What is already known about the subject? The knowledge may be local (e.g. from earlier audits, anecdotes, complaints) or more general and formal (e.g. from guidelines or performance indicators).

A wide variety of activities can be audited, for example screening, counselling, diagnosis, test requests, medical treatment, surgical interventions, follow-up, record keeping, communication, machine calibration and teamwork. Audit objectives should be expressed in down-to-earth terms reflecting how the problem appears in day-to-day practice.

Audit is not the answer to every problem. You should only spend time on an audit if you have a problem (neither the presence nor absence of a problem should ever be assumed) and audit appears to be the best way to deal with it.

Setting standards

Problems are identified in an absolute way or in a relative (comparative) way. Prescribing the wrong drug is generally considered unacceptable and by definition when it occurs it is a problem. In this case we are interested in each and every instance of the event, whether or not others have a higher or lower rate of events.

Most problems are defined in a relative way. For example, a 90% screening rate may not be considered a problem if similar practices achieve 85%, but it may give rise to concern if similar practices achieve 95%. Setting standards is a social rather than a scientific activity. You will need to exercise judgement as to what standard to set.

The principles of quality improvement suggest that for a relative problem you should not set a single final standard, but generate an interim standard (a stepping stone) to take you towards a final goal. The interim standard is then replaced with another, more stringent, interim standard, and so on until the goal is reached. Standards should be SMART (Specific Measurable Achievable Realistic and Time limited) and agreed by those whose practice is being observed.

Prioritising your time

Assuming that you feel that there is a problem worthy of addressing through an audit, you will need to prioritise your efforts. Again, this is an art rather than a science. Do not let an issue of concern loom larger simply because it is on your mind. It is worth standing back and deciding whether spending time on this problem is likely to yield better results than attending to something else.

Simple questions that you should ask are given in Box 1 (below).

Box 1: Questions to ask before embarking on an audit
  • Does the problem affect many people or just a few?
  • Is the problem minor or major for those affected?
  • Can you make a difference?
  • How much effort will it take to make a difference?
  • What will be the benefits to the practice of improving care (e.g. reduced time spent on administration as a result of improved communications)?
  • If practice does not improve, are there risks of unwelcome consequences (e.g. litigation, enquiry)?
  • Do your colleagues share your concern, to the extent that they are willing to put time and effort into addressing the issue? Can everyone agree fairly easily on how to define the problem and set standards?

Don’t reinvent the wheel

It is important not to reinvent the wheel. Most topics have been audited before. Where possible, try to ‘borrow’ other people’s good ideas and tweak them to suit local circumstances. Generally, colleagues are willing not only to let you see their paperwork, but to give you practical advice based on what worked for them – and what didn’t.

Going round the cycle

Plan the audit

There are four key areas that require attention to enable you to lay a sound foundation for an audit. You need to:

  • Ensure that the participants are committed to supporting the audit and willing to spend working time collecting data and reviewing the findings, and – this is often most difficult – spend time instituting any changes in practice that are required.
  • Consider carefully what data are needed to provide a useful perspective on your problem, given your objective. You should collect only data that are necessary, and collect all the data you need. Avoid the temptation to collect everything ‘just in case’. It is often useful to get help from someone who has experience of data collection, so that you can design the data collection forms in a way that minimises the time it takes and reduces the number of errors.
  • Define the sample and the data unambiguously. Again, you may find it useful to ask for help from someone who is experienced in survey methods. Ambiguity increases the time it takes to collect data and reduces the effectiveness of subsequent discussion.
  • Anticipate how you will want to present the data, so that data collection lends itself well to appropriate analysis.

Collect the data

If the audit is well planned, collecting the data should be easy. Sample sizes can usually be much smaller than people think. Successful audits can be paper-based. You do not have to create spreadsheets the size of Switzerland.

Organise the data

The data should be presented in a way that helps participants to see whether problems exist and to get a handle on what these might be. Questions likely to be raised during peer review should be anticipated.

Peer review

Participants should consider the data and ask:

  • Is there a gap between observed practice and the standards set?
  • If not, were the standards set correctly?
  • What are the possible reasons for any gaps observed?
  • What further data might be needed to clarify the problem?
  • What action should be taken, by whom, to improve the situation?

Taking action to change practice

Peer review should generate an explicit, focused, and time-bound action plan. It is essential to nominate an individual to make sure that the action plan is carried out. Progress should be considered at further peer review meetings. Failure to generate an action plan, or to carry it out, are very common reasons why audits do not achieve worthwhile results.

Re-audit

Once an action plan has been implemented, it is usually wise to carry out another audit to establish that practice has improved, because:

  • The problem may not have been identified correctly and the action plan may not, in fact, have changed the situation for the better.
  • People may have fallen back into their old habits once the initial interest in a problem has passed.
  • You may wish to raise the standard as part of an audit spiral towards the very best practice.

When to stop

You will only have time to deal with a very limited number of issues. You will need to decide when you can safely consider that the audit has come to an end. Unfortunately, this is not usually obvious: ideal practice has seldom been achieved when you may quite reasonably feel that the time has come to attend to something else. A simple checklist includes:

  • Are new ways of working firmly established?
  • Is the change maintained?
  • Have other areas of care become a higher priority for attention?

Considering when to stop an audit involves balancing the need to finish off the job properly with the need to start another one which is now high on the agenda. This is a matter for judgement, but resist the temptation to take on another project just because it is new.

Conclusion

Audit is a simple and practical tool. It is not principally about data collection, but about measuring current practice so as to understand where and how improvements can be made. There are methodological tricks to audit, but mostly it involves applied common sense and a commitment to being honest about current practice and making the effort to introduce changes.

Next month: Dr Russell Steele, Associate Director Post-graduate GP Education, South Western Deanery, discusses the audit submission requirements for summative assessment.

 

Guidelines in Practice, March 2003, Volume 6(3)
© 2003 MGP Ltd
further information | subscribe