The audit component of summative assessment provides an opportunity for GP registrars to learn an important skill for general practice. Dr Russell Steele explains how to get it right


 

The audit component of summative assessment is the training ground for a core part of GP life.

Audit is the method by which most GPs improve their standards and increase their income while still doing things that are relevant to everyday practice. It can be stimulating and demanding, and can introduce a competitive element between partners and practices, preventing stagnation and improving job satisfaction.

It is possible to pass the MRCGP or even win a research prize without demonstrating audit skills. However, GP registrars must show an understanding of audit to pass summative assessment. Some, who otherwise do well, fail the audit component simply because they have ignored, or not sought, advice on the subject.

Audit or review?

An audit should not be confused with a review. For example, a PCT may ask for figures that show how many patients with raised cholesterol are on statins, but if it does not ask for quality markers such as how many of them should be on statins this is not an audit. Rather it is a collection of facts with an implication that it is relevant to quality – a review.

Criteria and standards

The first commandment for GP registrars completing the summative assessment audit should be to use the headings given in the summative assessment guidance.

Audits in general practice should be about quality and improving quality. Quality cannot be assessed until the item to be measured has been defined. This is the criterion.

Audit criteria should be SMART:

  • Specific
  • Measurable
  • Achievable
  • Realistic (Relevant)
  • Time limited

For example, the criterion ‘patients with controlled hypertension will have their blood pressure checked’ is not SMART enough. A better one would be ‘patients with controlled hypertension will have their blood pressure checked at least once every 6 months’. Introducing the time element makes it more relevant.

The choice of criterion should be explained, even if it’s only ‘because it seems a good idea’ although it is helpful to justify your choice. References are not essential, but it helps to use them where possible because they often contain details that can justify both criteria and standards.

Once the criterion has been agreed the target should be set. The target, or success rate, is the standard for the audit.

Setting a target means deciding what success rate is achievable now, what may be possible in the future, and what is desirable. It may be desirable to achieve 100%, but this is not usually possible. On the other hand, a 10% success rate is not desirable either as it implies poor quality.

The standard should be set at a level that is challenging but not unreasonable. It could be quite high if the practice has a programme running already, but lower if the team has decided that the topic is worth working on, and wants to get better at it.

The reason for choosing a standard should be justified. Sometimes the justification is that the partners feel it is a standard they want to achieve, with no reference to the literature. However, if no-one has any idea about what standard to set, references may help by showing what other practices have achieved.

It is a good idea to involve the whole team when setting both criteria and standards.

The audit cycle

Effective medical audit is a cycle, or rather a spiral because each time round the cycle the standards are increased (Figure 1 below) unless the initial standards are set too high to achieve.

Figure 1: The audit cycle/spiral

Collecting data

Collecting data means finding out the current situation. This usually involves counting something, such as how many patients in the practice are on treatment for hypertension and how many of these had their blood pressure checked in the previous 6 months.

Analysing results

Analysing the results is a way of answering the questions, "How well did we do?” or, "Did we meet our target (standard)?”

Planning change

If the standard has not been met, it’s time to ask why and consider how it might be reached next time round. It does not matter if it has not been met as long as plans are made to improve in the next cycle.

Planning change when the standard has not been met involves finding out why it has not been achieved. Often it is because the standard was set too high, or because there was a problem agreeing the details of the audit.

Other questions to ask may include:

  • Are all those who need to be involved actually involved?
  • How can they become more involved?
  • Who has responsibility for taking action?
  • Would putting reminders in notes help?
  • Does the protocol need changing?
  • Is each step of the audit clear?

If the standard has been reached, the next stage is to think about how to improve. The standard should be raised for the next cycle and ideas for improvement discussed in the same way as when standards are not met.

After the changes have been discussed and agreed, the spiral continues with more data gathering, more analysis and more planning of change. Quality improves and audit becomes a comfortable part of everyday general practice.

How to pass summative assessment audit

A group of audit assessors in the South Western region found there were several common mistakes in the audits they assess. The advice in Box 1 (below) should help you avoid these errors.

Box 1: How to carry out a successful audit
  • Get started early
    • get some early drafts done for your trainer to look at
    • finalise plans for the audit during your first month
  • Keep it simple
    • no need for a complex audit with several criteria (one or two are enough)
    • the title should focus your mind
    • make sure it is simple, specific, relevant, measurable, justified
    • a good audit can pass with 1500 words
  • Keep it anonymous
    • make sure there are no references to your practice or partners
    • a simple practice demography is enough
  • Choose something that is likely to change
    • this makes the planning of change easier
    • summative assessment audits do not have to demonstrate change to pass
  • Justify your choice with references if you can
    • between one and five references are enough
    • it does not matter whether you use the Harvard or Vancouver system, but be consistent
    • assessors expect one or two key references
    • remember assessors may check the references
    • if you have no references explain why this is
  • Follow the marking schedule
    • there is no need to make up your own headings
    • remember the assessors use the marking schedule
    • using the headings on the marking schedule makes the audit easier to assess
  • Make sure the criteria and standards are explicit and clearly presented
    • there is frequent confusion about criteria and standards
    • criteria are what is being measured (e.g. number of patients with diabetes who have had HbA1c measured in the past 6 months)
    • standards are the targets aimed for (e.g. 85% of patients with diabetes will have had HbA1c measured in the past 6 months)
    • remember SMART (Specific, Measurable, Achievable in the time, Relevant to general practice, Time limited)
  • Involve relevant members of the primary healthcare team
    • explain how they were involved
  • Make sure data are laid out simply and accurately
    • explain tables and graphs
    • make sure numbers ‘add up’
    • cohort numbers do not have to be high – look on the summative assessment audit as a pilot
    • be sure similar cohorts are being compared in the second round
  • Make sure planned changes are properly explained
  • Ask yourself if it is reproducible
    • can someone else do the same audit if they read what you have written?

Together with the two commandments, following this advice will help you reduce your chances of failing or having to resubmit your summative assessment audit.

 

Guidelines in Practice, April 2003, Volume 6(4)
© 2003 MGP Ltd
further information | subscribe