Cochrane reviews of surgical treatments suffer from a lack of admissible evidence because very few high quality randomised controlled trials (RCTs) of surgical treatments are done. RCTs of surgical treatments are not done for various reasons:
Inherent limitations in conducting surgical trials are not prominently acknowledged in Cochrane Reviews. The search methodology, and the standards used to select or reject trials from inclusion in the review, are of course prominently featured, but little is said as to
“No evidence of benefit” is being misinterpreted as “this treatment does not work”. Evidence based commissioning of health services is shifting funding away from treatments which lack the “evidence based” imprimatur. This has serious health implications. Not only could effective surgical treatments be denied to those who need them, but the stepwise evolutionary development of surgical techniques (mainly resulting from the application of technological advances) risks being strangulated by regulatory and funding requirements to show evidence of effectiveness at each stage.
It is obvious to those of us who carry out mastoidectomy and tympanoplasty surgery for chronic suppurative otitis media that these operations, when carried out by competent surgeons, offer the best chance of definitive treatment for the patient. Although there are large published case series, there are no randomised controlled trials, so a Cochrane review is obliged to conclude “No evidence of benefit”. Although this particular review has now been withdrawn (to be replaced by seven separate reviews including one specifically on surgical interventions) the conclusion is misleading for patients and health commissioners. The argument can be illustrated by reductio ad absurdam:
Although no-one has yet published a Cochrane review of the incision and drainage of abscesses, I would predict the conclusion “no evidence of benefit” because there are no RCTs on the subject.
Where surgical treatment has a hard outcome measure - such as death versus survival in cancer or cardiac surgery – the choice of primary outcome measure for trials is clear. But most of the conditions we treat are not likely to result in death, we are operating to improve quality of life. Until recently, this was thought too difficult to measure, but the application of psychometric techniques to patients' symptoms in the 1980s began to allow a more quantitative approach to soft outcome measurement.
An explosion of research interest in the 1990s and early 21st Century has resulted in hundreds of disease-specific outcome measures, as well as numerous validated general health outcome measures. We are now spoilt for choice. Despite this, there remain significant difficulties in defining suitable outcome measures, particularly when it comes to variations in surgical technique.
In the protocol for a planned Cochrane review on the use of topical anaesthesia in flexible fibreoptic laryngoscopy, the clinical questions were
There was a clear difficulty in defining suitable outcome measures. Although “obtaining an adequate view” was the most important outcome measure, I commented that the definition of this was inherently subjective and operator-dependent. Furthermore, my personal standard would be “consistently obtaining an excellent view” and this standard could vary depending on the purpose of the examination. Secondary outcome measures would be patient discomfort and the incidence of side effects from the topical anaesthetic agents. Several parameters of the local anaesthetic preparation could be important, including
The selection of a particular technique is akin to the selection of a particular golf club for a particular shot. It is best left to the discretion of the expert. The more expert the golfer, the more likely he is to have the ability – and the requirement - to use a variety of different clubs, according to his professional assessment of the exact circumstances of each shot. Not only that, but another golfer with a different set of preferences might get just as good a result. It is therefore unlikely that any review based on analysis of RCTs could come up with useful generalisable recommendations.
Until recently, this didn't matter, because it was acknowledged that professionals knew best how to organise their specialist work. Now, however, managers and healthcare commissioners, ignorant of the subtleties and complexity of clinical expertise, will read the summaries of reviews and, armed with the assurance that there is "no robust and reliable evidence" to support one technique over another, will insist on standardisation to the cheapest. They congratulate themselves, convinced that they are "making the best use of scarce resources". Would they also like to tell Tiger Woods that, since there is no robust and reliable evidence for his preference, he has to use a limited selection from the cheapest golf clubs?
The desire of politicians and public to produce league tables has brought surgical skill into sharp public relief. Managers and health economists increasingly opine about the need to standardise “unexplained variances”. They seem to regard it as an unacceptable failing that 50% of surgeons are below average. An education department spokesman was once forced onto the defensive when confronted with the statistic that 50% of children were scoring below average marks in tests.
Skill is made up of a combination of inherent ability, which is developed by good training, then perfected and maintained by experience and regular practice. A surgeon who does not practice his skills, rather like a pianist who does not play, will tend to lose them. Some, less gifted with natural talent, will never be great no matter how much they practice. Some have had indifferent teachers, have never seen the finer points of their craft, and therefore aspire to nothing better.
Most established surgeons prefer familiar techniques, and, if the results are reasonable and predictable, carry on using them. Some surgeons are highly adaptable, migrate easily to new techniques, and are comfortable with a wide variety of approaches, while others are methodical "plodders", or become so with advancing years. A bad choice of technique has serious and immediate consequences for the patient - and the reputation of the surgeon. Preference for the tried and tested is, therefore, understandably commoner than desire to try the newest latest thing, yet advances rely on the latter.
This can be illustrated by another golfing analogy – the “Across the water” shot.
Judgement is something beyond skill. A surgeon with good judgement will tend to avoid getting into situations which test his advanced skills.
It must also be realised that surgeons do not work as isolated individuals, but as part of a team. Teamwork in the operating theatre is crucial. The finest and most skilled operator can have his performance ruined by an unfamiliar anaesthetist producing poor conditions, a clumsy assistant, a scrub nurse who doesn't know the instruments, bad maintenance of equipment, or a thousand other mishaps which can and will happen. In the UK, NHS surgeons have less and less control over these aspects of their performance environment. It is part of surgical judgement to know in which clinical environment to attempt certain cases. None of these factors can be controlled for in RCTs. They will not appear in any evidence base, and yet they are fundamental to successful surgical practice.
Modern surgery has developed primarily by the application of new technology. ENT as a speciality could not develop until the invention of the electric light bulb made reliable illumination of dark recesses possible. Improved visualisation – especially fibreoptic endoscopy and other forms of imaging – together with reliable, safe anaesthesia, and improved peri-operative care – have resulted in progressive improvements over the decades. Virtually no advances in surgery have been made by learning from the results of RCTs. Although there are some seismic changes – such as the introduction of cross sectional imaging in the early CT scanners of the 1970s - most of the improvements that occur in surgical techniques are incremental – a small improvement in an instrument, a slightly brighter light source, a higher resolution, finer fibre endoscope.
Although a number of golfing analogies appear in this article, I would like to make it clear that I am not a golfer. I gave up the unequal struggle to achieve consistent and reliable results 20 years ago, when it became clear to me that the amount of remedial training and practice required to compensate for my deficiency of natural talent in this area would considerably exceed the time available to a busy professional with a young family.
Evidence based medicine - historical perspective and critique
Philosophical, scientific and statistical basis of evidence based medicine
Popper KR 1959 The logic of scientific discovery. Hutchinson & Co, London. 8th Impession 1975 ISBN 0 09 111721 6
Cochrane AL 1972 Effectiveness And Efficiency: Random Reflections on Health Services. Facsimile Edn, additional contributions Silagy C, Chalmers I, 1999 RSM Press ISBN 185315394X
Sackett D.L., Straus S.E., Richardson W.S, Rosenberg W., Haynes R.B. Evidence Based Medicine: How to practice and teach EBM 2nd Edn 2000 Churchill Livingstone, London ISBN: 0 443 06240 4
All information and advice on this website is of a general nature and may not apply to you. There is no substitute for an individual consultation. We recommend that you see your General Practitioner if you would like to be referred.
© 2006 - 2008 All rights reserved
Mr James W. Fairley BSc MBBS FRCS MS (Lond)
GMC No. 2719566
Enquires: Mrs Sylvia Fairley RGN RCNT Dip Nurs (Lond)
tel 01233 642244 fax 01233 662840
Sandyhurst House Sandyhurst Lane Ashford Kent TN25 4NX
| We comply with the HONcode standard for trustworthy health information: verify here. |
|