Cochrane reviews of surgical treatments suffer from a lack of admissible evidence because very few high-quality randomised controlled trials (RCTs) of surgical treatments are performed. RCTs of surgical treatments are not performed for various reasons:
Inherent limitations in conducting surgical trials are not prominently acknowledged in Cochrane reviews. The search methodology, and the standards used to select or reject trials from inclusion in the review, are of course prominently featured, but little is said as to why there may be a dearth of acceptable evidence, and whether the required RCTs are ever likely to be done. Summaries and Conclusions are now easily available online to healthcare commissioners and the general public. ‘No evidence of benefit’ is being misinterpreted as ‘this treatment does not work’. Evidence-based commissioning of health services is shifting funding away from treatments which lack the ‘evidence-based’ imprimatur. This has serious health implications. Not only could effective surgical treatments be denied to those who need them, but the stepwise evolutionary development of surgical techniques (mainly resulting from the application of technological advances) risks being strangulated by regulatory and funding requirements to show evidence of effectiveness at each stage.
It is obvious to those of us who carry out mastoidectomy and tympanoplasty surgery for chronic suppurative otitis media that these operations, when carried out by competent surgeons, offer the best chance of definitive treatment for the patient. Although there are large published case series, there are no RCTs, so a Cochrane review is obliged to conclude ‘No evidence of benefit’. Although this particular review has now been withdrawn (to be replaced by seven separate reviews including one specifically on surgical interventions), the conclusion is misleading for patients and health commissioners. The argument can be illustrated by reductio ad absurdam: although no one has yet published a Cochrane review of the incision and drainage of abscesses, I would predict the conclusion ‘no evidence of benefit’ because there are no RCTs on the subject. There are no RCTs because surgeons know what to do with an abscess, and have known it for thousands of years – the benefit of treatment is obvious. In the case of the abscess, it is obvious to all. In the case of mastoidectomy and tympanoplasty, it is only obvious to the specialist in the field.
Where surgical treatment has a ‘hard’ outcome measure – such as death versus survival in cancer or cardiac surgery – the choice of primary outcome measure for trials is clear. I recently reviewed a protocol for a Cochrane review on the use of topical anaesthesia in flexible fibre-optic laryngoscopy. The question was whether topical anaesthesia was needed at all, and, if so, which would be the best agent. There was a clear difficulty in defining suitable outcome measures. Although ‘obtaining an adequate view’ was the most important outcome measure, I commented that the definition of this was inherently subjective and operator-dependent. Furthermore, my personal standard would be ‘consistently obtaining an excellent view’ and this standard could vary depending on the purpose of the examination. Secondary outcome measures would be patient discomfort and the incidence of side effects from the topical anaesthetic agents. Several parameters of the local anaesthetic preparation could be important, including rapidity of onset, degree of vasoconstriction/vasodilatation, depth of anaesthesia and duration of action. Different clinical circumstances might well dictate different preferences for these parameters. For a specialist working alone in an ‘office’ clinical environment, with the fibreoptic examination equipment immediately to hand in the same room, rapid onset and short duration of local anaesthesia would be preferable. For a specialist working in a typical busy NHS clinic, with several doctors sharing a treatment room, it would not be unusual to give the local anaesthetic spray and then find another doctor occupying the treatment room, and/or have to wait for the laryngoscope to be re-sterilised, while getting on with seeing the next patient. In that clinic, a longer duration of action would be preferable. Vasoconstriction would be an important attribute if the examination were to include an assessment of the nose and sinuses, but is virtually irrelevant in the context of a voice clinic concentrating on the larynx. In other words, there are horses for courses. In the main, I prefer to use 10% cocaine spray, but there are perfectly legitimate reasons why different specialists might prefer other techniques. Furthermore, the same specialist might use different techniques in different clinical circumstances. It is therefore unlikely that any review based on analysis of RCTs could come up with useful generalisable recommendations. Until recently, this did not matter, because it was acknowledged that professionals knew best how to organise their specialist work. Now, however, managers and healthcare commissioners, ignorant of the subtleties and complexity of clinical expertise, will read the summaries of reviews and, armed with the assurance that there is ‘no robust and reliable evidence’ to support one technique over another, will insist on standardisation to the cheapest. They congratulate themselves, convinced that they are ‘making the best use of scarce resources’. Would they also like to tell Tiger Woods that, as there is no robust and reliable evidence for his preference for golf clubs, he has to use a limited cheap selection?
The desire of politicians and public to produce league tables has brought surgical skill into sharp public relief. Managers and health economists increasingly opine about the need to standardise ‘unexplained variances’. They seem to regard it as an unacceptable failing that 50% of surgeons are below average. (An education department spokesman was once forced onto the defensive when confronted with the statistic that 50% of children were scoring below average marks in tests.) When setting up and interpreting RCTs of surgical techniques, the skill and preference of the surgeon become extremely important variables, and almost impossible to control. Skill and preference are not independent variables. Skill is made up of a combination of inherent ability, which is developed by good training, then perfected and maintained by experience and regular practice. A surgeon who does not practise his skills, rather like a pianist who does not play, will tend to lose them. Some, less gifted with natural talent, will never be great no matter how much they practise. Some have had indifferent teachers, have never seen the finer points of their craft, and therefore aspire to nothing better. Most established surgeons prefer familiar techniques, and, if the results are reasonable and predictable, carry on using them. Some surgeons are highly adaptable, migrate easily to new techniques, and are comfortable with a wide variety of approaches, while others are methodical ‘plodders’, or become so with advancing years. A bad choice of technique has serious and immediate consequences for the patient – and the reputation of the surgeon. Preference for the tried and tested is, therefore, understandably commoner than desire to try the newest latest thing, yet advances rely on the latter.
It must also be realised that surgeons do not work as isolated individuals, but as part of a team. Teamwork in the operating theatre is crucial. The finest and most skilled operator can have his performance ruined by an unfamiliar anaesthetist producing poor conditions, a clumsy assistant, a scrub nurse who does not know the instruments, bad maintenance of equipment, or a thousand other mishaps which can and will happen. In the UK, NHS surgeons have less and less control over these aspects of their performance environment. It is part of surgical judgement to know in which clinical environment to attempt certain cases. None of these factors can be controlled for in RCTs. They will not appear in any evidence base, and yet they are fundamental to successful surgical practice.
Modern surgery has developed primarily by the application of new technology. ENT as a speciality could not develop until the invention of the electric light bulb made reliable illumination of dark recesses possible. Improved visualisation – especially fibreoptic endoscopy and other forms of imaging – together with reliable, safe anaesthesia, and improved peri-operative care – have resulted in progressive improvements over the decades. Virtually no advances in surgery have been made by learning from the results of RCTs. Although there are some seismic changes – such as the introduction of cross-sectional imaging in the early CT scanners of the 1970s – most of the improvements that occur in surgical techniques are incremental – a small improvement in an instrument, a slightly brighter light source, a higher resolution, finer fibre endoscope. The improvement in quality is obvious to the skilled surgeon, who has the device in his hand, but not necessarily to others. Surgeons who are highly motivated to obtain the best results for their patients demand the best equipment. As skilled artisans, they know they can do a better job with better tools. If, however, they are obliged to produce ‘the evidence base’ for using this new and more expensive piece of kit, the evidence base is unlikely to be forthcoming. In order to demonstrate an improvement in clinical outcome to the standards required for a Cochrane review, trials involving large numbers of patients will need to be set up. This is likely to take several years, By the time the results are available, it is almost certain that the technology would have moved on.
Cochrane reviews of surgical treatments which conclude ‘no evidence of benefit’ due to a lack of high-quality RCTs should display, prominently in the plain language summary, a ‘health warning’ to non-experts interpreting the conclusions. This warning should include the following:
None declared.
Jim Fairley, Consultant ENT Surgeon
The following supplementary material is available for this article online:
Appendix S1. Extended manuscript.
This material is available as part of the online article from http://www.blackwell-synergy.com
© 2006 The Author. Journal compilation 2006 Blackwell Publishing Limited, Clinical Otolaryngology, 31, 496 - 498
HTML Version created for http://www.entkent.com December 16, 2007