The root causes of control-oriented failures
No plan survives contact with the enemy - Helmuth von Moltke the Elder
One must always consider a framework’s theoretical capability separately from its utility in execution - the latter is the proof of the pudding. Unfortunately, success for some stakeholders often comes at the expense of others, and when it does, resistance is inevitable, and trust in the endeavor is quickly eroded. Frameworks attempt to mitigate these risks, but their underlying methods are often not up to the task.
In Federalist paper #51, James Madison writes:
In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself. A dependence on the people is, no doubt, the primary control on the government; but experience has taught mankind the necessity of auxiliary precautions.
In The Square and the Tower, Niall Ferguson highlights further the weaknesses of people in these situations:
…the defect of autocracy is obvious, too. No individual, no matter how talented, has the capacity to contend with all the challenges of imperial governance, and almost none is able to resist the corrupting temptations of absolute power.
Sometimes failures arise as a result of defects injected by agents themselves. But too often, it is a failure to understand the weaknesses of the methods they use and the dynamics of the collective games that are being played. Let’s consider how frequent these problems arise.
A sampling of failures
The author of the book How Big Things Get Done, Bent Flyvbjerg (a professor at Aalborg University in Denmark) has been tracking over 16000 projects across 20+ different fields, and across 136 countries for over twenty years. His findings:
Only 8.5 percent of projects hit the mark on both cost and time. And a minuscule 0.5 percent nail cost, time, and benefits. Or to put that another way, 91.5 percent of projects go over budget, over schedule, or both. And 99.5 percent of projects go over budget, over schedule, under benefits, or some combination of these. Doing what you said you would do should be routine, or at least common. But it almost never happens.
Steve McConnell’s book More Effective Agile records his consulting business’s experience in over one thousand company engagements with Agile endeavors. As you can see, the performing organizations - on average - failed to apply Agile key practices consistently and effectively across this landscape, even when blessed with the benefits of experience from experts. Unless there is a ‘closed loop’ approach to such practices, this is exactly what one might expect given the inevitable cultural resistance that accompanies introducing these practices into routine practice.
Several large endeavors are particularly noteworthy across the landscape of my experience. Such endeavors tend to be ambitious projects that fear delivering reduced capabilities which can erode trust and accountability in their sponsors, providers, and methods. When ‘selling’ the endeavor, the underlying mindset is to offer optimistic projections drawn from limited surveys while downplaying risks and uncertainty. Their common maladies typically involve overreach and limited learning:
Transportation
Puget Sound’s mass transit project, Sound Transit - according to Bloomberg - was to be a showcase that would show that providing better service would attract more riders and generate a virtuous cycle of growth. The current endeavor was approved by 54 percent of voters in 2016 and was to include sixty-two miles of light rail throughout the region, to be completed by 2041; the delivery was phased, and construction was well underway when the Bloomberg article was written in 2018. It has not gone well.
One of the key segments was across the I-90 bridge, which moves with wind, water, and traffic. In 2022, it was discovered much of the bridge installation required extensive demolition and rebuilding (much as other segments have encountered, suffering of all things rail gauge incompatibilities). These issues pushed the original 2023 opening of cross-bridge traffic to 2026. The original $55B budget, which voters (paying over $1700 per household per year in new taxes) were assured had more than adequate contingency for any uncertainty, is now looking at between a $30-40B overrun. Five other segments to suburbs have also suffered delays of up to seven years, while their taxes, and additional overruns, continue. The current system carries less than 1% of all trips in the region, and has failed to offload overcrowded roads, as promised.
An updated plan is expected in the next year, but such problems, per the previous link, are always viewed as financial, with the expected solutions additional revenue and deferred implementation. Worse, revenues from riders have tanked, as the adopted hub-and-spoke design reduces rider convenience and its transfers lengthen end-to-end times. Luckily, self-driving cars are beginning to get approval for ride sharing in limited areas, and by the time the Sound Transit plans are completed, this more effective and less costly alternative will likely be ubiquitous.But it’s hardly the worst transportation endeavor currently in work. Consider California’s High-speed rail effort, authorized by a 2008 statewide initiative to deliver continuous service between San Francisco and Los Angeles with a nonstop travel time of one hundred sixty minutes. Phase 1 (five hundred miles long) was to be delivered by 2020, at a cost of $33B. Phase 1 is expected to share tracks with conventional trains, operating at a higher average rate than France’s fastest TGV trains. They were warned this was infeasible back in 2008. Costs to date have tripled and projected completion is thirteen years after original commitments. It’s no wonder that funding is questionable since the initial construction package will only connect Fresno and Bakersfield, over which little traffic is expected.
How’s NASA doing in transporting astronauts to outer space? We are no longer hitching rides on Russia launch capability, but our astronauts still need a place to ride. The Orion Capsule is NASA’s crew spacecraft capable of supporting a crew of four beyond low earth orbit and has been under development since 2006. Its specifications call for twenty-one days of undocked service and six months of docked service, and are to be used by the Artemis program, NASA’s current lunar exploration endeavor. Orion has already spent $31B (2025 dollars) and struggles to recreate the mission first accomplished over sixty years ago (the total expenditures including the Space Launch System approach $100B). The maladies of the doomed effort are well chronicled here; suffice to say that if only cost and schedule were the problem, one could still kindle some hope, but it’s far, far worse than that. Meanwhile, SpaceX’s private space efforts continue to astound, with a vision pointing to establish a permanent Mars settlement, rather than completing another trip to the moon.
Government services
Health care
The U.S. health care system is innovative but fiscally unsustainable. It delivers cutting-edge treatments and short wait times for specialized care, but at the cost of uneven access and worse population health outcomes than other wealthy nations. These problems have worsened since the launch of the Affordable Care Act, which has relied on escalating infusions of subsidies, rather than effective cost control.
Education
Since the founding of the Department of Education
$3 trillion spent
Reading scores for 13-year-olds have not improved
Only 27% of 8th graders proficient in math
Only 31% of 8th graders are proficient in reading
47 cents of every federal education dollar spent on compliance
If you’re defending the Department of Education, you’re defending a failure. For example, five years ago, according to a recent report from UC San Diego faculty, only about 30 students arrived with math skills below high-school level. Now, that number is now more than 900, with most failing to fully meet middle-school math standards. As a result, the students supported by these investments are expected to earn 8% less than graduates 12 years ago.
These examples demonstrate the broad reach of failures to learn how to affordably deliver across the landscape of applying traditional frameworks to current endeavors.
Reality always requires investments in understanding
Each existing framework has limited explicit contextual guidance on features critical to endeavors, instead focusing on prescriptive content definitions for expected artifacts (many of which are rarely read or maintained). A better strategy would instead provide practical guidance on how to apply critical thinking skills to inevitable tradeoffs while advising and reinforcing behaviors necessary for success. For example, planning is indeed essential, but its utility can evaporate once problems dictate deviations which are outside the range of the original plans. It is challenging to refactor (rather than patch) the details, especially when such changes impact work in process, and when little thought has been given in advance to how adaptability will be employed by the endeavor itself.
When the endeavor is large, complex, and ‘too big to fail’, political will inevitably contaminates prudent judgement, a recipe for failure rather than success. There are always plenty of excuses - environmental reviews, inflationary costs, adjustments in scope, externalities - but all typically arise because the basis of forecasts were biased and presumed the plans in place were based upon sound assumptions rather than incomplete knowledge. Once these assumptions inevitably unravel in long-lived projects, those guilty of overreach may have moved on to their next adventures, and a new wave of leaders emerge to claim that a comprehensive reset (with new assumptions) will curtail further sins - if stakeholders will just wait a while longer and shell out a lot more. Until they won’t.