7
The Trouble with Ratings
We have seen how surgeons can improve their mortality record by actually killing more patients. With a little expertise, manipulation, and perhaps some malice, a less good surgeon may be rated higher than a better one. Sadly, however, this is not the only reason why league tables can be misleading. They can mislead with no malice at all. All it takes is a little ignorance.
One of the few legacies that Margaret Thatcher’s government left on the NHS was some degree of fiscal probity and accountability. She and her ministers were keen that hospitals did their accounts appropriately, were able to explain income and expenditure, and learned to budget properly. To some extent, this was successful. When Tony Blair’s Labour government took office, it shifted the emphasis from pennies to outcomes, and wanted to measure how well hospitals were doing things. The Department of Health picked a handful of conditions, and asked hospitals to measure their outcomes for these conditions. One of them was heart attacks.
We have seen in Chapter Two that a heart attack occurs when a narrowed coronary artery gets completely blocked, usually by a fresh clot that forms on a furred-up area (or plaque) in the wall of artery. As the clot blocks the blood flow down the artery, the part of the heart muscle that that artery was feeding simply dies. The patient gets a terrible chest pain that doesn’t get better with the usual measures. The heart may fail if the piece of muscle that dies is big enough. There may be dangerous disturbances in the rhythm of the heart, or the heart may stop completely. The bit of dead muscle may give way, causing a rupture with severe bleeding in the chest, or a hole may form in the heart, or a valve may give way and start to leak very badly. It all depends on where (and how big) the dead bit of heart muscle is. Not surprisingly, some of these complications can be fatal, and that is why people sometimes die from heart attacks. Most, fortunately, do not. They are admitted into a coronary care unit, where they receive treatment to prevent and manage all of these complications, and doctors try to reduce the size of any dead bit by re-opening the artery (if you can get at it fast enough).
Heart attacks are common, and survival is an objective and reasonable outcome for measuring how well they are treated, which is why the UK government was interested in them. So hospitals were ranked according to their individual heart attack mortality, and, in the interest of transparency, the results were made available to the public: a heart attack league table was born. My own hospital, Papworth, was the absolute worst, with a mortality of 40 per cent for heart attack, whereas most had a mortality of between 10 and 15 per cent. It was appalling news. Patients awaiting heart surgery were calling us in a panic, and the hospital management team thought that we had a real problem on our hands.
We began to wonder where this 40 per cent figure came from. Normally, at the time of this report, patients with heart attacks were admitted to acute general hospitals. These are hospitals with Accident & Emergency departments equipped to deal with most emergencies, and having a coronary care unit specifically to admit patients with heart attacks. At the time of that league table, we were not an acute hospital, and we did not have a coronary care unit. In other words, we simply did not admit patients with acute heart attacks: they went to two other general hospitals nearby. A brief search into our data promptly revealed what was going on. The reason we had such a ‘high mortality from heart attack’ was not that we were bad, but that the only patients we treated with this condition were those who came to us in extremis from the two other nearby hospitals and from other general hospitals in our region.
Rarely, the part of the heart that dies in a heart attack is the muscle between the two pumping chambers, or ventricles, of the heart. Even more rarely, the dead muscle gives way, creating a defect or hole, with blood going round in circles within the heart and lungs instead of nourishing the body. The condition is called ventricular septal rupture. When that happens, the patient becomes very sick indeed, rapidly progressing into heart, lung, and kidney failure. Death is usually inevitable unless the defect is closed surgically. It is, of course, a very high-risk operation carried out urgently on a sick heart that has not yet recovered from the ravages of a big heart attack, so it is not surprising that it is an operation with very high mortality. Up to about half of the patients who have this operation die from it, but at least it offers some chance of survival, so it is better than doing nothing. When doctors in a coronary care unit in one of the hospitals around us make the diagnosis of ventricular septal rupture, they normally refer the patient to us for emergency surgery.
In the period covered by the heart attack league table, there were five such patients with ventricular septal rupture, and they were the only ‘heart attack’ patients we had treated. Two died: 40 per cent, which is actually a pretty good result. We managed, eventually, to re-assure our other patients and their relatives that they were, in fact, safe to be treated at our hospital, but the cost to us and to our patients of this ‘transparency’ was substantial.
We live in an age where transparency is a buzzword. Politicians and the media never tire of asking for more of the stuff. Predictably, their enthusiasm for transparency rapidly seems to fizzle out when they are asked to provide some clarity about their own, often sordid, affairs. Regardless of their behaviour, we lesser mortals providing public services are nowadays expected to provide a transparent insight into our work patterns and outcomes, and the field of healthcare is no exception. If those who work in healthcare generally are not immune from constant demands for more transparency, then those specialties in which good data are available are much more susceptible than others. After all, it is pretty difficult actually to demand open access to data from a specialty that does not collect any data. For that reason, my own specialty of cardiac surgery has found itself an easy target in the firing line of the ‘more-transparency-now’ big guns.
After the public outrage that followed the Bristol Royal Infirmary affair, heart surgeons found themselves in the unenviable position of having lost much of the public’s confidence in them. One of the recommendations of the Bristol inquiry was to push hard for transparency and the publication of results, and the Department of Health at the time had made this issue a priority. Somewhat reluctantly, with many dissenters and loud interminable arguments at national specialty meetings, the cardiac surgical profession agreed to work towards the publication of outcomes, and this has now happened. If you go to www.scts.org and follow the links ‘Patients’ and ‘Heart surgery in the UK’, you will find web pages with access to the data from every heart surgery hospital in the country, with survival figures for overall heart surgery and certain specific operations, most of which will have been risk-adjusted using EuroSCORE or a more stringent version of EuroSCORE, and most hospitals will have provided data subdivided by individual consultant surgeons. You can, if you wish, scrutinise my own outcomes for heart surgery. In terms of transparency, heart surgery has already achieved much more than other areas of medicine, and the UK has achieved much more than other countries.
Yet the journey towards this level of transparency was not without pitfalls. On 1 January 2005, the Freedom of Information Act came into force in the UK. In grossly simplified terms, the act mandates that information collected by public bodies using publicly provided resources should be made available within 20 days to any member of the public who requests it. Shortly before the act became law, the chief executive of Papworth Hospital received a letter from The Guardian newspaper, informing him that it was the paper’s intention to submit an official request under the act to provide the data on the hospital’s mortality for coronary artery bypass grafting (CABG). He asked me what we should do, and the answer to my mind was obvious: we should, of course, comply. We had the data, we were proud of the data, and it was the law! A quick check with colleagues elsewhere in the country confirmed that the chief executives of their hospitals had also received a letter from the newspaper, and some were quite perturbed by this sudden demand.
The Guardian, however, did not ask about risk profiles, so we guessed that the plan was to publish a league table for CABG with no attention to risk. This would mean that the listed figures would be misleading if the purpose was to inform the public. As we have already seen, when the mortality for CABG is 2 per cent in hospital X but only 1 per cent in hospital Y, there can be three possible reasons for the difference:
Reason 1: the difference is due to chance.
Reason 2: the difference is due to a different case mix (hospital X operates on many high-risk patients, and hospital Y operates on low-risk patients).
Reason 3: hospital Y is better.
Before we can conclude that hospital Y is better than hospital X, we need to know the confidence intervals around the measurements (in other words, whether the difference is real or due to chance). Once that is established, we need to know the case mix of the patients (if the case mix in both hospitals is similar, hospital Y is truly better than hospital X). The Guardian had not asked for this information, and we felt that they should have asked, so we contacted them. Two days later, the two reporters working on this project came up from London to Cambridge to find out more. I explained to them the issues of statistical confidence and risk profiling, the use of risk models such as EuroSCORE, and the importance of taking account of these sorts of factors when reporting or publishing health outcomes. They took it all on board.
The article appeared on Wednesday 15 March 2005, as a front-page headline story backed by two whole pages in the main paper. The newspaper published the results for hospitals across the UK. Some were crude, some were risk-adjusted, and most were by hospital and surgeon. The data were divided into risk-adjusted and non-risk-adjusted, listed alphabetically, and with intelligent explanations about the results, the caveats needed in interpreting them, and the importance of statistical analysis and risk profiling. In other words, the paper behaved with integrity and responsibility, and avoided all temptation to sensationalise its findings.
Other newspapers have reported the same issues with far more lurid copy. If ten patients had died in one hospital and 30 in another, this would have been reported as ‘SCORES OF PATIENTS ARE DYING NEEDLESSLY IN THE UK’. In the British Medical Journal, Dr Jan Poloniecki drew attention to other potential yet meaningless shock-horror headlines that could be paraphrased as ‘HALF OF THE SURGEONS IN THE UK ARE BELOW AVERAGE’. This, of course, is absolutely true. It simply happens to be what the word ‘average’ means: half are above, and half are below. If they weren’t, average wouldn’t be average. Sensational and sloppy reporting of this nature can result in patients and the public getting the wrong message altogether, with unnecessary and distressing loss of confidence affecting perfectly competent, world-class health services. It is not, however, only the less savoury sections of the media that can thus mislead. Even the professionals can get it wrong.
Dr Foster is an organisation that took it upon itself to make hospital information available to the public. It is a professional outfit, with experts in both medicine and statistics on board. In the early days of its operation, it of course concentrated on cardiac surgery, presumably because the data were available and relatively easy to interpret, and the outcome measure (survival) was objective and obtainable. Dr Foster began to collect data by relying on information from Hospital Episode Statistics, or HES. This is a system in which clerks assign codes to patients when they leave the hospital. These codes include procedures performed on the patients during their hospital stay. Dr Foster thus identified from these data which patients had CABG and only CABG, and reported on the mortality. To this day, I vividly remember sitting, with fellow cardiac surgeons in the august hall of the Royal College of Surgeons in London to listen to eminent Dr Foster officials explain to us what data they had, how they acquired and analysed the data, and how they were going to make everything public in the interests of transparency.
HES data are notoriously inaccurate. The codes are chosen by poorly paid clerks, some of whom may not have had enough training in interpreting medical case notes. The medical case notes themselves are often incomplete. So I did not have a lot of confidence in the accuracy of whatever Dr Foster officials were going to tell us, but I nevertheless listened with an open mind. The spokesman said that they had taken great care in collating the data and ensuring that only CABG patients were included in their analysis. He assured the audience that Dr Foster were meticulous in identifying patients who had CABG plus ‘something else’, and took pains to exclude such patients from the data. ‘Something else’? We had assumed that Dr Foster would remove patients who had CABG plus, say, a heart valve operation, or CABG plus replacement of the aorta, and so on. The spokesman went on to assure us that all of the patients who had CABG plus below-knee amputation were absolutely excluded from the analysis.
Excuse me? Below-knee amputation?
Now that is a most unusual combination, to say the least. The commonest isolated operation in cardiac surgery is CABG. The commonest combined operation is CABG plus replacement of the aortic valve. Of course, there are many other combinations of operations, but for a patient to walk into hospital one day and, a few days later, hobble out with a coronary artery bypass graft but minus a leg must be vanishingly rare. What was Dr Foster doing, looking at this category specifically to exclude it, and to re-assure us that it was excluded from analysis? From that point onwards, I stopped paying attention to the slick presentation on the podium, and my mind wandered as I was trying to figure out why on Earth a patient would come into hospital for a CABG and a leg amputation in the first place. Try as I might, I could not think of a credible clinical scenario. Then I remembered that Dr Foster studies HES data. These data record the codes for what actually happened during the hospital episode, rather than what the patient originally came into hospital for. Illumination finally dawned.
The confounding factor was a marvellous invention called the intra-aortic balloon pump, or IABP. This clever device is one of the few medical gadgets that could be described as a true lifesaver. It is a pump that helps the ailing heart, especially if the heart is being starved of blood supply and oxygen, because of narrowed arteries. The IABP can, in specialist units, be connected to the patient and activated in about 10 to 20 minutes at the bedside. The doctor puts a needle in the femoral artery, then threads a wire through the needle so that the wire comes up the femoral artery into the descending aorta, which courses along the spine. The wire stops when it reaches the top of the aorta, in the chest. Over this wire, a specially designed long sausage-shaped balloon is inserted so as to lie within the aorta, some 20 centimetres downstream of the heart. The balloon is connected to the pump, and the pump can read the patient’s heart beat. When activated, the pump rapidly inflates the balloon with helium while the heart is relaxing, and rapidly sucks out the helium and flattens the balloon when the heart is pumping, so that it follows the action of the heart, but beats out-of-synch with it. This provides two huge advantages: the first is that, by deflating the balloon just as the heart is about to start pumping, it cuts down the pressure in the aorta, making the heart’s job much easier (it is easier to pump blood into a low-pressure system than a high-pressure one). The second is that, by inflating and raising the blood pressure when the heart is relaxing, it forces blood down the coronary arteries, improving the blood supply to a heart that may be starved of oxygen. An IABP can get a patient out of heart failure better, quicker, and more effectively than any drug, and patients whose angina is so bad that they are teetering on the edge of a full-blown heart attack find the angina instantly relieved as they are pulled back from the brink by the IABP. One of the most satisfying procedures in medicine is setting up the IABP in an awake patient. The instant that the pump is switched on, the patient smiles, and thanks the doctor for getting rid of the angina pain.
Such a machine is a godsend to heart surgeons, especially those doing CABG on critical patients. If the heart is not working too well at the end of the operation, the IABP buys time, gives the heart a rest, and allows it to recover over the next couple of days. Yet in surgery, as in everything else, there is no such thing as a free lunch. IABPs have their complications, and one of those complications relates to the point at which the balloon is inserted, and where it stays during the course of the treatment: it is, of course, the femoral artery in the leg. If the femoral artery is damaged by the procedure, or bleeds, or fills up with clot, or is small and gets blocked by the sheer physical size of the balloon, then the leg is at risk. Very rarely, the leg is damaged beyond salvation, and a below-knee amputation is carried out.
That is why Dr Foster found the strange combination of below-knee amputation and CABG. These were standard CABG operations that went badly wrong, needed an IABP, which itself went badly wrong, and lost the hapless patient a leg. And what did Dr Foster want to do with these patients in their league tables? Why, exclude them of course, thinking they were not standard isolated CABG. Yet such patients absolutely must be included in any league table looking at the success and failure of CABG, because they are precisely the ones where things went horribly wrong. The problem was that the highly intelligent folk at Dr Foster simply did not know that.
Since that particular episode, Dr Foster have improved massively in their understanding of the data that they analyse and report, and I do not think that errors quite as glaring as this would happen today in their reporting. Nevertheless, this example illustrates yet another danger of publishing league tables when the data from which they are derived are not analysed intelligently.
Heart surgeons and their patients are now accustomed to the publication of outcomes, but other specialties are not. They will have to learn fast: on 28 June 2013, the NHS began publishing the outcomes of surgery in a non-cardiac surgical specialty. For the first time ever, the results of vascular surgery (surgery on blood vessels) were being made available in the public domain, and those of other surgical specialties will follow. Many of these non-cardiac surgical specialties do not, as yet, have sophisticated risk-assessment models, and the data are therefore even more likely to be misread and misinterpreted than in heart surgery. Transparency is, of course, a good thing, and league tables of results may indeed inform the public about the outcomes of healthcare, so that an individual patient and his or her family can make an informed choice, but there are many, many dangers for the unwary.
I have shown some examples where reported numbers can be meaningless without appropriate statistical analysis, and other examples where comparisons can mislead and even cause panic because they are not comparing things that are comparable. I have also shown how a bad surgeon can look better than a good surgeon in a league table, and given some examples where vital data can be missed out altogether in the analysis so that entire league tables are misleading. Finally, I hope to have convinced you that patients and doctors alike can be damaged by irresponsible and half-baked sensationalist reporting by those sections of the media keener on headlines and profit than on truth.
That, unfortunately, is not all.
The greatest risk to the patient in the publication of league tables is that the surgeons start to run away from high-risk surgery, and that is bad news indeed, especially if you are such a patient.
The coronary artery bypass graft operation is probably the most studied, scrutinised, researched, and reported therapeutic intervention in the history of medicine. Compared with most therapies, our knowledge about CABG is truly massive. We know who will benefit in terms of relief of symptoms. We know who is likely to live longer as a result of CABG, and who is likely to die if not offered a CABG. We know the likely risk of the operation to a relatively high degree of precision, and can tailor a custom-made quote for each individual patient having it. We also know the risk of not having CABG, and can compare the two. We have a pretty good idea of how long a CABG will last, and which of your three bypasses in a triple CABG will probably continue to function right up to the day you die of something else, such as being run over by a bus.
We also know that patients with weak hearts are more likely to die if they get a CABG than patients with strong hearts, but, despite that, other things being equal, patients with weak hearts should be even keener to go under the knife and receive a CABG. The reason for this is simple: if CABG is a little riskier for the weak-hearted, then not having a CABG is much, much worse. This has often been called the cardiac surgical paradox, and it can be stated simply, and brutally, like this: the more the operation is likely to kill you, the better it is for you.
This strange maxim does not simply apply to CABG, but also to many other areas of heart surgery. It sounds paradoxical, yet it is fairly easy to understand why it is true. By and large, simple operations done for simple things in the heart are low-risk, but so are the conditions that they treat, whereas complex, horrendous operations are done for complex, horrendous conditions, and these conditions are more likely to kill if left untreated. Recall the somewhat extreme condition of ventricular septal rupture. It is almost 100 per cent fatal if untreated, with a 40–50 per cent mortality rate if treated. Which would you rather have?
That brings us to the problem. Let us imagine a typical heart surgeon. He is now subject to transparency and close scrutiny, and his results are published on the web for all to see. He is on call one weekend when he is asked to see an 80-year-old female patient with a tired, weak heart. She has just had a heart attack, and had such bad angina afterwards that she had to be connected to an IABP. He looks at the results of her tests and sees that she has critical coronary disease, of the kind that leads rapidly to death without a CABG operation. He also sees that, after a long history of smoking and bronchitis, her lungs are not too great. The blood tests show that her kidneys are a tad dodgy as well. More worryingly, though, she is having chest pain despite being in bed, and even despite the medication and the IABP, meaning that she truly is on the brink of another heart attack, which will probably be fatal, so his only choice, if he is to do anything, is to operate now, as an emergency.
Being an educated and cautious surgeon, he calculates the risk of surgery using EuroSCORE II, the latest version of the risk calculator, and it tells him that the risk of her dying from surgery is 38 per cent. Without surgery, however, there is no way out for the patient. She will probably die from a heart attack soon, and even immediately if the IABP, which has kept her hanging on to dear life so far, is removed.
Our surgeon, however, is feeling rather good about himself. He has had a pretty good run so far. It is 27 March, four days before the end of the British financial year, in which he has done 72 CABG operations without a single death. His figures for this year will look amazing in the league tables. If he operates and she dies, his mortality jumps from zero to 1.4 per cent, which is roughly the national average. He is now faced with a dilemma. Surgery might be good for the patient, but may be bad for him, his figures, his private practice, and his naturally large ego.
Do league tables really deter surgeons from accepting high-risk patients? I recently surveyed the cardiac surgeons in the UK to find out. I asked them two simple questions. The questions, and the answers given by the cardiac surgeons, are given below:
A high-risk operation may be beneficial to a particular patient. Despite this, a cardiac surgeon may decide not to offer that option to the patient, and recommends continuing medical treatment. This is partly or wholly because of concern about the impact on that surgeon’s figures should the patient opt for surgery and then succumb. Have you ever done this?
Are you aware of other surgeons doing this?
Of the 115 surgeons who responded to the survey questions, 35 (just under a third) admitted to denying surgery to patients who may benefit because of concern about their figures, and the great majority (84 per cent) reported that they were aware of other surgeons doing the same thing.
This survey has confirmed without any doubt that the clinical decision-making process of surgeons has indeed been adversely affected by the culture of transparency. Most surgeons who responded to the survey reported that they had seen such risk-averse behaviour in their colleagues, and nearly a third were honest enough to have reported such behaviour in themselves. The best interests of the patient, which should always come first, have had to take second place to the league tables.
With a health economist’s hat on, one could argue that healthcare systems should probably not be offering high-risk surgery to anyone. The fact that such surgery is expensive, that resources are limited, and that the risk makes the operation less beneficial than in a young, otherwise fit person would seem to back this view, but we are forgetting two important considerations when we follow such a train of thought. The first is that doctors are primarily here to do the best for their patients (and not for the healthcare budget or their league-table figures). The second is the cardiac surgical paradox: the more an operation is likely to kill you, the better it is for you. This is simply because, in many such patients, withholding an operation has such a dismal outcome that the risk is more than amply justified. Yet withholding an operation and similar risk-averse behaviour unfortunately happens all too frequently. It is another reason why we should approach league tables with great care and circumspection. Sometimes, the price of transparency is unacceptable. Our poor old patient, if she is refused an operation, will die and become a statistic, but not a statistic that appears in the league table of heart surgery, and nobody will ever know.
Transparency is here to stay, and nothing we can do will turn the clock back. We can mitigate some of its damage by ensuring that the data are as accurate as possible, that all published data are properly adjusted for risk, that the analysis is statistically sound, and that the data are presented intelligently, but no matter what we do, we can never get away from the fact that, for a surgeon, the easiest and most effective way to reduce mortality is to say ‘No’ to high-risk patients, that the temptation to do so is a strong one, and that many surgeons will yield to it. There are, however, two things we can do. The first is to introduce a structure that allows surgeons to take on very high-risk patients without fear of damaging their own career and professional status, as illustrated by something called the Star Chamber.
When it comes to operating on very high-risk patients, the following facts are apposite. First, the cardiac surgical paradox often means that the patient, provided he or she survives, will stand to benefit enormously from the operation, and, if an operation is not carried out, the outlook is pretty bleak. Second, the surgeon, if the patient survives, will get kudos and feel good, but if the patient dies, will suffer physically, psychologically, and professionally. For a surgeon with even the tiniest scintilla of humanity, few things are worse than losing a patient. I know how I feel about it when it happens, and I have spoken at length about it to friends and colleagues. We all deal with the aftermath in different ways, but one feature of such a loss is common to all of us, and that is an overwhelming feeling of desperate loneliness. After all, the buck stops nowhere else but at our very door, if you will forgive the mixed metaphor. Third, in any particular group of surgeons, it tends to be the same ones who will take on the very high-risk patient, and who therefore have to take the ‘hit’ for everyone else if the patient dies. All of these are problems, but they can be turned into an opportunity.
At Papworth, we have always tended towards being one of those hospitals that accepts patients at the high end of the risk spectrum. National figures show that our patients are the oldest and have the sickest hearts in the country, and these differences are statistically significant. Even if we compare our patients with those of the hospital that accepts the second most elderly and sick patients, we also find the difference to be statistically significant. In other words, compared to the rest of the country, when it comes to how sick and high-risk the patients are, we are out on a limb: a true outlier.
As the culture of transparency and league tables began to take hold in cardiac surgery, our local motley crew of surgeons started to experience a certain feeling of unease about our particular type of high-risk practice, and this unease lay at the fork of a dilemma. We could proceed in one of two ways.
We could carry on in our merry way to take all comers, regardless of risk, if we felt we could do them some good, but that would lay us open to ostracism should some of us begin to look bad in the league tables of surgeons. Even within Papworth, it tends to be a select group of us who are inclined to take on the very highest risk patients, and this is due to a combination of experience, specialised skills in certain operations, a pronounced desire to do the best for the patient regardless of risk, and, let’s face it, sheer bloody-minded arrogance and self-confidence. When that is taken into consideration, we can see that this particular handful of surgeons would suffer more ostracism than most.
The alternative approach would be for us collectively to begin to say ‘No’ to the highest risk patients, like so many other surgeons and hospitals do already. The problem with this approach is that it would mean denying patients what could be their only chance, the loss of our hard-earned reputation as the hospital where ‘anything is possible’, and, last but not least, a serious dent to our collective professional ego.
That is when we came up with the concept of the Star Chamber. We never called it that. In fact, it went by the more prosaic title of the Surgical Council, and that is still its official title, but while I was explaining the concept to Bruce Rosengard, an American surgeon who worked at Papworth briefly, he exclaimed, ‘Holy cow, you mean like a Star Chamber?’, and the unofficial title stuck fast.
The original Star Chamber was an English court of law that sat at the royal Palace of Westminster from the 15th to the 17th century. Its intended purpose was to enforce the law against prominent, rich, and powerful people, who appeared immune to the efforts of the ordinary courts. The court of the Star Chamber sat in secret, with no indictments, no witnesses, no juries, and no right of appeal, and dispensed justice the way it saw fit. The court eventually became a somewhat dubious political weapon, used and abused by the monarchy. The idea formed the basis of a 1983 Hollywood film, also called The Star Chamber and set in modern times. In the movie, an idealistic vigilante judge finds ways of dispensing rough justice to any criminals who, in the view of the court, are guilty yet manage to escape punishment on a technicality. In the film version, the Star Chamber dispatches a hired killer to mete out capital punishment to those it finds guilty.
Needless to say, our Star Chamber did none of this, and its aim was certainly not to kill folk, but to offer an operation to those who would be at risk of being turned down by surgeons who are looking over their own shoulders at the league tables. It works very simply: any surgeon who is referred a patient considered to be at exceptionally high risk is encouraged to use the option of calling a meeting of the Star Chamber. Such patients may include those turned down by other hospitals, or those with a very high EuroSCORE (25 per cent or above), or any patient that the surgeon feels is exceptional in presenting a very difficult and high-risk challenge. The Chamber consists of all the consultant surgeons, but needs a minimum of four surgeons to attend in order to be quorate. The patient’s case is then discussed in detail, and the Chamber makes three decisions: whether or not to offer surgery, the nature and strategy of the operation, and who in the group is best placed to do it, with the proviso that two consultant surgeons will be involved. This pair will perform the operation on behalf of the group, and the group will collectively take the full responsibility for the outcome.
On the face of it, the advantages of such an approach are legion. First, patients who would benefit from an operation will actually get their chance. Second, one would hope that the pair of surgeons who are best at this particular type of operation would be the ones selected to do it. Third, those selected to do it are working in the name of the group, and do not have to worry about their figures. And finally, the input of a group is always useful for covering all the bases (four heads being better than one), and that could mean a wiser, safer, and more carefully planned operation.
So far, the Star Chamber experiment has been a qualified success. On the positive side, many patients have been assessed through the system, and over half were offered operations, and most of these were successful. On the negative side, I feel sure that not all patients who are suitable have benefited from the approach, so that there may be some who were turned down without being considered, and others who were accepted by individual surgeons without consulting the Star Chamber. Interestingly, patients referred from elsewhere to the Star Chamber tended to do very well, whereas those brought by Papworth’s own surgeons did less well, showing that it really takes a lot for a Papworth surgeon to be sufficiently frightened by a patient’s level of risk to come to the Chamber.
Nevertheless, the Star Chamber continues to run at Papworth Hospital, providing a valuable service to some of the sickest heart patients in the country.
The second solution to the problems of risk-avoiding behaviour is to shift the emphasis in quality control from transparency onto a different, more robust and less harmful system.
Transparency means that outcomes are made public. It does not necessarily mean that they should be presented as a league table. The Guardian newspaper, for instance, was able to report all the data obtained from UK heart hospitals under the Freedom of Information Act without once succumbing to the temptation of presenting the results as a league table. On the contrary, the results were reported by hospital in alphabetical order, with separate subsections for those that used a risk-scoring method and those that did not. Other newspapers and sections of the media may not be so responsible, and the sensationalist lure of the hit parade is difficult to resist for an editor and a headline writer hungry for sales.
Let us pretend that you are a patient contemplating having a CABG somewhere in the UK, and you are presented with the following:
|
Institution |
CABG Mortality |
|
St Elsewhere Hospital |
0.9% |
|
Holby General Infirmary |
1.2% |
|
The Shire University Hospital |
1.5% |
|
The Heart Clinic |
1.9% |
|
Ambridge Royal Hospital |
2.3% |
|
Ambridge Surgeons |
|
|
Dr Green |
1.8% |
|
Dr Turquoise |
2.1% |
|
Dr Purple |
2.4% |
|
Dr Brown |
2.7% |
How likely is it that you will choose Ambridge Royal Hospital? And how likely is it that you will choose Dr Brown? Probably not at all likely. Yet Dr Brown and Ambridge Royal may well be statistically no better or worse than anyone else. They may even operate on higher-risk patients, and their results may actually be among the best in the country when the risk profile of their patients is taken into account. Nevertheless, I suspect most patients would run a mile if they saw the positions of Dr Brown and Ambridge Royal in the league table.
Let us now add to this toxic cocktail of potential misinformation a small dash of sensationalism: a newspaper reports the league tables not as an exercise in information and transparency, but as a shock-horror story, so that the front page is plastered with pictures of Dr Brown, carefully and deliberately chosen to make him look mean and unfriendly, along with an insert photograph of his suburban mansion on the outskirts of Borchester, with the inevitable million-pound price tag, another picture of his Porsche with its personalised number plate, and a headline screaming ‘IS THIS THE WORST SURGEON IN BRITAIN?’
Apart from the inherent potential of misleading the public, league tables carry the substantial risk of putting whoever is the hapless surgeon or unfortunate hospital at the bottom of the league out of business altogether, so that if a newspaper publishes an account similar to the above, a surgeon’s career may come to an end. When that happens, the surgeon stops appearing in the league table. Now that Dr Brown has gone, attention will focus on Dr Purple, who is now ‘the worst surgeon in Britain’, and the same will happen again. Taken to its logical conclusion, this process will eventually leave only one surgeon operating in the country. This is patently absurd. What is important for patients and their loved ones is not necessarily to have the ‘best’ surgeon in the ‘best’ hospital, but to know that the local hospital and surgeon that treat them by providing surgery are of a high standard that is considered up to scratch by modern criteria. In short, the patient needs and deserves an absolute guarantee that the surgical service treating him or her is one of quality. To that end, what we need is a system of quality accreditation.
The principle of quality accreditation is a very simple one. Hospitals should have in place a robust structure that ensures the following:
Now, these five requirements are not rocket science. They are easily within the capability of all hospitals doing surgery, and all they require is some rudimentary data collection and a little bit of paperwork. Indeed, I have argued that if you can’t tell that your outcomes are up to scratch, you have no business doing heart surgery, or any other kind of surgery.
The European Association for Cardio-Thoracic Surgery, the European Society of Thoracic Surgeons, and the European Society for CardioVascular Surgery are the three biggest speciality societies in the fields of heart, lung, and vascular (blood-vessel) surgery in Europe. Some ten years ago, the three got together and established a body called ECTSIA, the European Cardiovascular and Thoracic Surgery Institute of Accreditation, which was empowered with offering official certificates of accreditation along the above lines to hospitals that were proven to satisfy these modest criteria and wanted their achievement recognised. To date, in all of Europe, the total number of hospitals that have applied and succeeded in obtaining this accreditation amounts to the grand sum of … three!
Perhaps many hospitals satisfy the criteria, but did not take the steps to achieve official recognition. Perhaps the initiative was not advertised widely enough, but this is unlikely to be the reason: I happen to know that when this quality accreditation plan was announced, no fewer than 500 different hospitals in Europe enquired about it and expressed an interest. From 500 to three is a huge drop. Could it be that even these simple structures are still not well developed enough for hospitals to take the plunge?
Building a system of proper quality accreditation of the type described above is a lot less ‘sexy’ than the transparency culture of publishing results, and attracts very little media interest, but it can ensure that robust quality-assurance and monitoring mechanisms are integrally woven into the very fabric of the way a hospital functions, and it can provide all patients with the peace of mind that comes with the knowledge that their surgery providers meet the acceptable standard. To use the American cliché, what’s not to like?