Triaging unwarranted variability
Uncontrolled variation is the enemy of quality - Deming
After going to all the trouble of standardizing our work, and training our people, it is easy to expect consistent outcomes as the fruits of our labors. In practice, variation prevails until we can eliminate its causes.
Consider Figure 1, which is an artist's depiction of the frequency and position of urban crimes in South London, Manchester, East London, and Eindhoven. Each individual occurrence of violent crime in each area adds to the height of the model at that location. The result forms a mountainous terrain of the variation which the locations have in these crime incidents. Such variation is the nature of statistical populations and how they differ, even in similar environmental conditions. This variation is a challenge to solution designers who are working to achieve the greatest good for such populations.
For an example drawn from the health care domain, consider this situation, discussed in Health Dialog:
If you live in northern Idaho, and you develop back pain, chances are good that you’ll undergo surgery to treat your pain. Move to the southern tip of Texas, however, and the chances that you’ll undergo that same surgery will drop by a factor of 6. The surgery is no more effective in Idaho than it is in Texas. It’s just that doctors in the northwest are more likely than those in southern Texas to recommend surgery. This phenomenon, in which doctors practice medicine differently depending on where they’re from, is called practice pattern variation. And it isn’t limited to treating back pain, or even surgical decisions.
The above situation is called unwarranted variation. It is only one of many distinct types of such variability, which arises from many diverse sources:
the presenting symptoms and their underlying root causes (which affects the frequency and complexity of encountering such situations)
the presenting conditions of the subject (i.e. medical history, the severity of illness, previous treatments, receptiveness to recommendations, etc.)
the novelty of the situation, and the resulting ability to use experience and draw from relevant knowledge during treatment
the individual skill and diligence of the treatment team
slips (making the proper decision regarding treatment, but making one of many missteps along the path of implementing that treatment) and mistakes (making the wrong decisions in particular situations)
We'd like to think that outcomes will be uniform, except for a few unusual cases. What we find instead is that distributions of outcomes follow one of several statistical distributions, with a handful of quite poor outcomes, a handful of exceptional outcomes (some that may appear miraculous), and a broad, undistinguished middle ground.
For example, according to the book Better:
After an ordinary hernia operation, the chances a patient will have a recurrent hernia are one in ten in the lower group, one in twenty in the middle, and under one in five hundred for the top performers.
For newborns admitted to Neonatal Intensive Care Units, the risk-adjusted death rate can vary between 6 and 16 percent, depending upon the hospital.
The likelihood of successful pregnancy for in vitro fertilization varies from under 15 percent to over 65 percent, depending on where you go.
Within a particular diagnosis, differences of type 2 from the above list, combined with variations in the willingness of an institution to accept high-risk patients for treatment, will account for some of these differences, but many are simply due to the defect types 3-5, above. Over time, many of the sources of this variation can be identified and systematically reduced, but no member of the medical team is free from mistakes and slips; diagnostic errors average about 10%; errors in reading radiology films are well over twice that.
As described by Gawande:
In medicine, we are used to confronting failure; all doctors have unforeseen deaths and complications. What we’re not used to doing is comparing our records of success and failure with those of our peers. I am a surgeon in a department that is, our members like to believe, one of the best in the country. But the truth is that we have had no reliable evidence about whether we’re as good as we think we are. Baseball teams have win-loss records. Businesses have quarterly earnings reports. What about doctors?...
What one really wants to know is how we perform in typical circumstances—some kind of score for the immediate results, perhaps, and also a measure of the processes involved. For patients with pneumonia, how often does my hospital get them the correct antibiotic, and on the whole how do they do? How do our results compare with those of other hospitals? Gathering this kind of data can be difficult. Medicine still relies heavily on paper records, so to collect the information you have to send people to either scour the charts or track the patients themselves, both of which are expensive and laborious propositions.
Let's explore one disease, cystic fibrosis. This disease is infrequent, occurring in about 1000 patients per year, and given its low frequency (about 1 in every 3800 births), it is remarkable that we have made the progress that we have. Much of what we know and have accomplished with this disease is due to the diligence of the Cystic Fibrosis Foundation, which has been supporting research and patients for over 50 years. It has invested in countless research studies, helped decode the gene for the disease, and has underwritten countless innovations - new therapies, drugs, and devices. All of these could not have been done by themselves, but required a partnership with dedicated practitioners, working in a synergistic relationship. And their collective record of accomplishment of successes has been astonishing.
In 1957, the average patient with CF died by age 3. In 1964, the CFF gave a pediatrician at the University of Minnesota, Warren Warwick, ten thousand dollars to collect data on every patient at the thirty-one CF centers that year. Warwick discovered an example of positive deviance, a physician named Leroy Mathews at the Babies and Children’s Hospital in Cleveland. Leroy Mathew's patients had a median age at death that year of twenty-one years. Turning back to the book Better:
Unlike pediatricians elsewhere, Matthews viewed CF not as a sudden affliction but as a cumulative disease and provided aggressive preventive treatment to stave it off long before his patients became visibly sick from it. He made his patients sleep each night in a plastic tent filled with a continuous aerosolized water mist so dense you could barely see through it. This thinned the tenacious mucus that clogged their airways, enabling them to cough it up. Using an innovation of British pediatricians, he also had family members clap on the children’s chests daily to help loosen the mucus. After Warwick’s report came out, Matthews’s treatment quickly became the standard in this country. The American Thoracic Society endorsed his approach, and Warwick’s data registry on treatment centers proved so useful that the Cystic Fibrosis Foundation has continued it ever since. Looking at the data over time is both fascinating and disconcerting. By 1966, mortality from CF nationally had dropped so much that the average life expectancy of CF patients had already reached ten years. By 1972, it was eighteen years—a rapid and remarkable transformation.
At the same time, though, Matthews’s center had got even better. The foundation never identified individual centers in its data; to ensure participation, it guaranteed anonymity. But Matthews’s center published its results. By the early 1970s, 95 percent of patients who had gone there before severe lung disease set in were living past their eighteenth birthday. There was a bell curve, and the spread had narrowed a little. Yet every time the average moved up, Matthews and a few others somehow managed to stay ahead of the pack. In 2003, life expectancy with CF had risen to thirty-three years nationally, but at the best center it was more than forty-seven.
Today, cystic fibrosis patients get care at one of 117 specialized centers across the country, each having undergone a rigorous certification process, which assures that they follow the same guidelines for treatment, (which are far more detailed than for most other diseases). All these centers support ongoing research to find new and better treatments. Yet differences across treatment centers continue to be dramatic, especially considering the exemplar at the University of Minnesota. Returning, again, to Better:
The director of Fairview-University Children’s Hospital’s cystic fibrosis center for almost forty years has been none other than Warren Warwick, the pediatrician who had conducted the study of LeRoy Matthews’s suspiciously high success rate. Ever since then, Warwick has made a study of what it takes to do better than everyone else. The secret, he insists, is simple, and he learned it from Matthews: you do whatever you can to keep your patients’ lungs as open as possible. Patients with CF at Fairview got the same things that patients everywhere got—some nebulized treatments to loosen secretions and unclog passageways (a kind of mist tent in a mouth pipe), antibiotics, and a good thumping on their chests every day. Yet, somehow, everything Warwick did was different. Warwick's combination of focus, aggressiveness, and inventiveness is what makes him extraordinary. He thinks hard about his patients, he pushes them, and he does not hesitate to improvise.
Like most other medical clinics, the Minnesota Cystic Fibrosis Center has several physicians and many more staff members. Warwick established a weekly meeting to review everyone’s care for their patients, and he insists on a degree of uniformity that clinicians usually find intolerable. Some chafe. He can have, as one of the doctors put it, "somewhat of an absence of, um, collegial respect for different care plans." And although he stepped down as director of the center in 1999, to let a protégé, Carlos Milla, take over, he remains its guiding spirit. He and his colleagues aren’t content if their patients’ lung function is 80 percent of normal, or even 90 percent. They aim for 100 percent—or better. Almost 10 percent of the children at his center get supplemental feedings through a latex tube surgically inserted into their stomachs, simply because, by Warwick’s standards, they were not gaining enough weight. There’s no published research showing that you need to do this. But not a single child or teenager at the center has died in almost a decade. Its oldest patient is now sixty-seven. In medicine, we have learned to appreciate the danger of ad hoc experimentation on patients—of cowboy physicians. We endeavor to stick to established findings. But with his unblinking focus on his patients’ actual results, Warwick has been able to innovate successfully. And he has become almost contemptuous of established findings. National clinical guidelines for care are, he says, "a record of the past, and little more—they should have an expiration date."
This example highlights a dilemma of systematic improvement: If no one is free to innovate, or has the time to do so, little will change; but if everyone innovates independently, there will be no collective learning or systematic improvement. The key is to eliminate the common causes of variation efficiently and progressively over time and isolate those from the special causes - the errors that arise in execution, which can be tackled with training and increased oversight and diligence. There are two key practices for this to happen - transparency of outcomes and reliable measures of effectiveness.