• drives or shapes your subsequent actions over a planning horizon (typically multi-year)

• are not easily changed once made, and

• will have the greatest impact at this point in time on your long-term success

This handful of decisions includes:

• selecting your strategic posture with respect to your customers for the products and services which you intend to offer them

• identifying the value propositions which you will pursue with those customers through a coherent concept of operations

• developing approaches to create or align projects with these sources of value

• constructing tailored delivery systems to realize benefits from these projects as soon and broadly as possible

In synthesizing these strategies over time, customer-driven value should be pursued by extending:

1. Speed: the ability to deliver results faster than before, and respond rapidly to the emerging needs of the business

   • Approaches to quickly and successfully launch activities for the most frequently occurring customer engagement scenarios

   • The ability to rapidly and reliably reallocate resources to new situations with the minimum of friction

   • Responsive and effective infrastructure for satisfying operational service requests

2. Capacity: assuring a supply of products and services that is sufficient to meet the needs of all customers

   • Capacity that is scalable to meet current and future requirements cost effectively (up and down)

   • Strategies to extend the reach of products and services so that unit costs can be driven down

3. Efficiency: More products for less resources across the life cycle of your products and services

   • Managing assets and execution resource consumption to extend the leverage of investments

   • Eliminating chronic patterns of waste

4. Delivery: Products and services that accelerate delivery of results to the customer

   • Architectures that enable responsive, cost-effective solutions for customer’s needs - now and in the future

   • Scalable solutions which offer value and choices to the customer for different levels of investment

   • Approaches which insulate customers from the underlying means of production (‘handle the details’)

5. Quality: Reliable realization and cost-effective satisfaction of quality expectations for the customer

   • Assuring that the quality of products and services is adequate to support customer needs

   • Coordinating infrastructure changes so they have with minimal impact to customers

   • Keeping systems operating at peak performance

   • Ensuring that infrastructure provides security controls and maintains the integrity of the organization’s information

   • Managing corrective action and restoring services when outages or problems occur

Opportunities to pursue such value will only be worthwhile if they are appealing to your customers, can be monetized, and if the resulting revenue or resource authorizations can be redistributed among the organizations to reimburse the costs of implementing the required changes, and sustain and support the changes over time. All of these require anticipating the customer’s needs based upon insights derived from working together over time and developing an understanding of their preferred patterns, chronic issues, and future direction.

Regardless of how they are communicated, key success criteria for evaluating and selecting them are as follows:

1. They must promote a consistent understanding of their intent, scope, and reach across multiple stakeholders

2. They must integrate and exploit the conceptual knowledge available within the organization

3. They must provide utility for the organization’s business and its evolving environment under multiple possible future scenarios

4. They must facilitate exploration and trade-offs of candidate approaches to implement the strategies as the environment evolves over time

1. All work must be clearly defined as to content, sequence, timing, and outcome.

2. Every customer-supplier connection must be direct, explicit, and transparent

3. An effective protocol must be used to communicate requests and information about deliveries and priorities.

4. The pathway for delivering every product and service must be simple, seamless, and well understood by the responsible agents.

5. Improvements must be guided through carefully controlled experiments, under the guidance of an experienced coach, at have ownership at the lowest possible levels in the responsible groups.

Lean means the elimination of waste. In manufacturing, this typically means reduction of material in inventory, since that material is expensive, and since such reductions cause efficiencies in other important aspects, such as cycle time. In knowledge-based industries, waste is not extra people, as you might expect, but rather a reduction in the information and decisions that are ‘in inventory’, or essentially, in processing. This waste reduction is usually further augmented with additional efficiency improvements, by balancing capacity against demand and prioritizing work queues.

Lean practices have not yet achieved the penetration within engineering groups that they have achieved in manufacturing. I believe this is for several reasons:

1. Engineers don’t always understand why cycle time is important to the business.

2. A lack of a clear and brief explanation of how lean principles can be applied to the engineering environment.

3. There haven’t yet been enough good examples and ‘how-to’ guidance available or shared on Engineering Lean practices that have been relevant to engineering fields.

As a result, many engineering groups think Lean practices are not relevant to what they do, believe that Lean is a way of thinking, rather than a collection of techniques, or do not know what steps they might take to employ these techniques.

In manufacturing environments, variation typically arises when recurring, mechanistic operations are implemented inconsistently. Within engineering, such routine operations do not exist, since each design problem is unique, requiring searches through unfamiliar territory for potential solution elements.

Identifying waste

What’s common to both environments is a mandate to improve, and Lean can help in that regard. To accelerate outcomes, waste is anything that stands in the way of that objective. Pursuing waste reduction unlocks access to the next goal - removing these obstacles from the pathway of future endeavors.

For example, if regular meetings don’t seem to be helping move towards these goals, then in the spirit of removing waste, the design of these meetings should be reconsidered. Streamline the communications path by separating those critical for decision-making from those who merely want to be kept informed.

Avoiding rework, upping quality, and root-cause analysis

If a feature has an unusually high defect density, it may contribute to wasted time and effort. This wasted time doesn’t help in getting software in the hands of the customers, therefore, an effort should be made to reclaim it. The only way to do that would be to - get it right the first time. To do that, one has to ensure that no (or as few as possible) bugs are found during the final testing, instead, everything ought to be caught and fixed earlier.

This means two things - one, that testing needs to be done early and often through all stages of the game. Two, each time a non-trivial issue is discovered, the root causes should be investigated to prevent the same kind of thing from happening again. This is the equivalent of the ‘pull the Andon cord’ philosophy of Lean manufacturing.

Getting better at the difficult stuff

One reason why certain teams to achieve frequent releases is because deployment is hard. It can be an arduous process with lots of moving parts. That is the kind of situation that is ripe for fresh eyes and new thinking. Thet technical debt that inevitably piles up when not doing this boldly, simply must be avoided at all costs - else it will kill even the possibility of speedy delivery.

Multi-skilled people

When tasks require unique skills, those skills can become single points of failure should those individuals be unavailable for any reason. Such bottlenecks must be resolved. Having redundancy of skills across a Lean team can help mitigate this risk.

A culture of excellence

A culture of excellence can contribute to reducing cycle-time further. These include limiting the size of queues, achieving single-piece flow, using pull scheduling systems, and proactively discovering the next opportunity that can improve throughput of the team.

Each team-member should have a caring attitude for the customer and a curious attitude. One set of questions can be directed towards the customers themselves, to gain a fuller understanding of what they’re trying to achieve, so that the best, most creative, and efficient solution can be found. Another set of questions can be directed to the team and its processes. By following these practices, things become better for the customer, for the company, and for the team as well.

This culture of excellence judges its results by one metric alone: throughput. It realizes that any step taken to improve things should be measuring its effects. This culture should focus on how to optimize the whole system, and not just a part of it. Cycle-time is one of the easiest and most effective measures for managing this throughput, and can be enhanced by following these practices.

Once offerings are placed into service, existing resources need to begin be allocated to improve the quality and functionality of the product which may have been viable, but is now on life support. These improvements become the focus of subsequent iterative cycles in order to refine, shape, and extend the value of the solution for a broader range of customers. As figure 1 shows, the size of the batches packaging this value has a direct relationship on the holding costs for the business, which are a function of the availability of the information, material, and resources necessary for future investments.

As an example, in Managing the Design Factory, Don Reinertsen highlights how such batch size tradeoffs can play out in protracted test cycles:

Let us define the batch size of our testing process to be the amount of design work that we complete before we start testing. The largest batch size would be waiting for the entire design to be complete before we start the test. This will minimize the number of tests, but we will discover our defects very late in the design process, when it is expensive to react to them. In contrast, if we test a partially complete design, we deliver the information associated with the test earlier. Our problem is to balance the cost of extra test cycles against the cost of making expensive changes late in the design cycle.

Faster iterations enable multiple efforts to search for value in parallel with the decision cycles framed by their circumstances. Timeboxing of these iterations facilitates the exploration of more alternatives within a given timeframe and provides opportunities for more effective evaluations to be performed earlier, to determine which ones are worth saving, so the rest can be pruned out without a lot of collateral damage. These adaptation tactics form an essential practice for businesses which enables convergence of fitness for use through incremental cycles of learning. This convergence is effective even when performed within a constantly changing environment, as long as the team’s agility is faster than the pace of environmental change.

In the initial iterative cycle, the initial conditions, transformations, and context may all be flawed, but it is necessary to be able to tell the difference. We would like to be able to reduce uncertainty and improve our projections of the work to go until our original commitments will be achieved; the earlier and more frequently we obtain information about that target and our path towards it, the better off we will be.

As products come and go, those that glitter may not subsequently be found to contain gold. The evaluation of opportunities must be adequately framed, so that evaluation and prioritization can be performed competently. Implementation then requires a series of transformations across multiple planning horizons, disciplines, organizations, product breakdown structures, and work breakdown structures. Control of batch sizes is critical to enable feedback to be incorporated with each cycle of these transformations, unlocking tremendous efficiencies:

Author Rose George explores the taken-for-granted environment of containerization around the world, and its often opaque ownership structures, regulations, and inhumane working conditions. The world of container ships has everything sized in terms of twenty-foot equivalent units. Ninety percent of our modern world's consumption travels by these means, and make it possible to send a can of beer around the world for about a penny. As the book describes, this creates odd incentives which make it cheaper catch fish in Scotland, freeze them and ship them to China, filleted them, re-freeze them, and shipped them back to Scotland, where they are sold in a Scottish grocery store at less than had the processing all have been done locally by Scottish workers.

Erhardt Reichlin, author of The Art of Systems Architecting, describes how to achieve this progressive elaboration - by reducing batches and increasing iterations - as follows:

Stepwise refinement is the progressive removal of abstraction in models, evaluation criteria, and goals. It is accompanied by an increase in the specificity and volume of information recorded about the system, and a flow of work from generalized to specialized design disciplines. Within the design disciplines the pattern repeats as disciplinary objectives and requirements are converted into the models of form of that discipline. In practice, the process is neither so smooth or continuous. It is better characterized as episodic, with episodes of abstraction reduction alternating with episodes of reflection and purpose expansion.

Since the value of information uncovered during this pursuit quickly decays, it must be acted on quickly. While more information seems preferable to less, in practice, information is only trusted in decision-making when it is accurate, meaningful, and relevant to the situation. The groups responsible for accomplishing this stepwise refinement need sufficient capacity to respond to latent demand while having effective channels to return work packages that are not sufficiently developed for realization.

• who are the customers?

• what do they need in order for business objectives to be realized?

• what functions must be provided by solutions to fulfill these needs?

• how well must solutions perform for these functions to meet expectations for cost, quality, and schedule?

• which requirements are necessary to characterize acceptable solutions in order to focus and manage development?

• how will the uncertainty of achieving these requirements within the negotiated resource bargain be managed?

Here is Stanford Beer on such resource bargains:

The Resource Bargain is the ‘deal’ by which some degree of autonomy is agreed between the Senior Management and its junior counterparts. The bargain declares: out of all the activities that System One elements might undertake, THESE will be tackled (and not THOSE), and the resources negotiated to those ends will be provided.

However autocratic or democratic (or even anarchic) your Resource Bargaining proves to be, the governing mode of management is Accountability. Please think about this responsibility for resources provided in terms (not of financial probity, not of emotional dependency, but) of variety engineering. Can you possibly itemize every single thing that the subsidiary does, demand a report on it, and expect a justification? Obviously not. Therefore accountability is an attenuation of high-variety happenings [and] investment is a variety attenuator.

EXAMINE precisely how accountability is exercised and especially what attenuators (totals, averages, key indicators) are used. If in the end, you are appalled to discover that the machinery is inadequate, that Senior Management just does not have Requisite Variety, then you had better own up.

The discipline of triage is essential to close any gaps discovered between the identified problem and solution spaces and establish situational awareness for aligning the viewpoints of design, production, and support systems. Gaps have different levels of importance for different stakeholders, and these stakeholders have different interests at different points in their system’s lifecycle.

OODA loops have proven to be effective constructs to shape a cyclic set of activities for organizing the pursuit of agility over these cycles, especially in high-pressure environments. Chet Richards positioned this framework as a strategic framework to progressively align actions to changing environments: “Strategy is a mental tapestry of changing intentions for harmonizing and focusing our efforts as a basis for realizing some aim or purpose in an unfolding and often unforeseen world of many bewildering events and many contending interests.”

Each cycle of iteration of these transformations conceptually can be divided into 4 phases:

• Orientation

  Orientation begins by performing a situational assessment which involves the exploration and framing of the perceptions of actors, the states of objects they are focused on, and the significance of events which shape those states, as each object interacts within a broader context. The goal should be to develop a neutral understanding of the collective intentions of stakeholders relative to the environment and conditions which present themselves. It is usually helpful to use a set of organizing questions to help shape the collection of this information in order to establish connections between actions and consequences, by projecting not only what is happening, but how the situation is likely to unfold over time under different scenarios.

• Observation

  Given this orientation, observations are collected by actively acquiring information from available sources. In philosophical terms, it is the process of filtering sensory information through a mental process. In science, observation additionally involves the recording of data via the use of instruments. Nearly all observations will be a mixture of objective and subjective assessments; these observations should include context, learnings, and an innate “feel” for situations, people, and events. It should be recognized that each of these is influenced by a mix of variables, some within the subject’s control, and others outside it. The underlying goal of both activities is to synthesize a mental operational definition that is grounded in facts and whose interpretation (or meaning) is consistent with observations. There is sometimes a question in data processing as to where “observing” ends and “drawing conclusions” begins, though the boundary typically involves recognizing intensions, and evaluating options for aligning behaviors and intentions. Casual analysis can also be helpful, if time permits.

• Decision-making

  Decisions require appropriate framing in order to make balanced trade-offs between target outcomes, time, resources, and risk. A major part of this decision-making involves identification and analysis of a finite set of alternatives that can be evaluated using a limited set of criteria The speed within which decisions must be made is likely to determine the decision process involved; decision methods such as that used by the military are thorough but time-consuming. When the observer is the decision-maker, this task might vary according to whether the decision must be made intuitively or intellectually. If the latter, one must rank these alternatives in terms of how they are perceived to impact target outcomes, both in isolation and as interactions between the criteria when all are considered simultaneously. When decisions instead are to be made by a group, adoption of a more formal decision process should be considered.

• Taking Action

  Actions are necessary to establish momentum and begin achieving progress towards objectives. In military operations, the objectives are sent as an operations order. The sooner one can begin getting started, the faster one can begin learning. Of course, one can’t initiate just any action; it must be purposeful, conscious, and focused on making progress. Tyler Cowen stresses the importance of taking action even in the face of limited data: “The more information that’s out there, the greater the returns from just being willing to sit down and apply yourself. Information isn’t what’s scarce; it’s the willingness to do something with it.”

As Kyle Eschenroeder says, “we need to rebalance our appreciation for the power of action with our tendency to overthink, over-plan, and otherwise waste our energies in abstraction.” The work of Fernando Flores has been instrumental in guiding how to transform an existing chain of transactions into a lean and efficient set of organizational capabilities through learning. The pace of this learning depends on how quickly and effectively provider and customer concerns about the necessity and sufficiency of provided capabilities can be fulfilled:

A process can be improved if it can be understood; it can be understood only if it has a consistent structure [Senge 1990]; and its structure can be consistent only after the first steps of process improvement have reduced process variability... Many processes exhibited such broad variation in behavior that it was difficult for process specifiers to agree on a process that represented the typical scenario. Many organizations informally built process specifications from anecdotal process experience instead of developing the baseline process model with empirical models and data. Many organizations... created an ideal specification instead of capturing empirical practices, and organizations used these specifications as a baseline for improvement despite the mismatch. Because many process specification models were divorced from empirical practice, they could not reliably drive real development practice.

Organizations can track the effectiveness of their decision-making by measuring the momentum at which benefits are being realized from investments, rather than by measuring the velocity of the teams who are doing the work. Preparation for this realization capture requires orientation of both the context of customer concerns and the utility of solutions being deployed by service providers.

In Stafford Beer’s Viable System Model, the multinode is the structural and cognitive configuration of the organization’s “identity function” when decision-making is not the result of a single person but a collective mind. It is Beer’s way of describing how multiple individuals can jointly enact the role of ultimate policy, coherence, and long‑term direction. His comments on this notion illuminate convergence:

We claim we know how the whole thing works. The problem is to make it work more quickly. That must surely mean the introduction of discipline and order, of some sort, into the situation. It also means, however, that no measures may be adopted which would at the same time put the remarkable freedom of action and the wonderful flexibility of the multinode in jeopardy. If people could see how to do this, without putting themselves and their organization into a strait-jacket, there is some chance that they would adopt new techniques. One method, we ought now to agree, must be excluded although it is the one method most usually attempted in practice, because no one can think of anything else. This is the method of rigorous protocol. Explicitly: it denatures the system itself - with all its in-built capacity to generate the right answers.

The first difficulty is to know what kind of problem the multinode actually solves. It does not devote itself, its seniority and power, to the determination of trivial outcomes or it ought not to do so. It is likely to be settling a policy of great importance and therefore considerable complexity. Thus it is that people think of thinking as a process of synthesizing an integral but elaborate conclusion from a large number of component parts. The decision is seen as a rococo edifice built up clause by legal clause. This is perhaps why there are endless drafting problems facing anyone trying to promulgate an agreed decision.

The cybernetician adopts the contrary position. The output of the thinking process, the decision, has the following form: do this (rather than anything else). When the process of thinking originally starts, the multinode is faced not indeed with a number of building bricks in an edifice but with a seemingly infinite number of possible outcomes between which it must choose. It is the existence of this plethora of possibilities that cries out for decision in the first place. Then, under this model, the process we seek to assist is one of chopping down ambiguities and uncertainties until we may say: do this. In short, we would like to measure the variety of the complex decision at the start, measure the reduction of variety brought about by each conclusion reached in the process of thinking things out, and in general monitor the entire operation of the multinode as the variety comes down to a value of the decision itself. To do this we shall need two tools: a paradigm of logical search, and an actual metric - a rule and a scale - for measuring uncertainty.

The speed of this search is the critical discriminator between groups that can deliver sustainable, exemplary performance and those who just get by. Those in the latter class are implicitly condemned to a death by a thousand cuts.

Operational excellence is not a single methodology or a set of best practices; it’s a discipline of coherence: the ongoing ability of an endeavor to align intent, capability, and execution so that work flows smoothly from concept to completion. At its core, operational excellence depends on three things:

• Clarity of purpose (what the system is trying to achieve)

• Visibility of work (how value is actually produced)

• Coherence of behaviors (how agents act within the system)

When these elements reinforce each other, endeavors develop a kind of operational gravity: work moves in the right direction with less friction, fewer surprises, and more resilience. But this gravity only emerges when the underlying orientations of the people doing the work are recognized and integrated rather than left to operate in isolation.

The Role of Agent Orientations

Every agent in a system operates from a viewpoint. Some are Process‑oriented, optimizing flow and reducing variation. Others are Insight‑oriented, probing for meaning, patterns, and strategic leverage. Still others are Action‑oriented, driving momentum and execution. These orientations are inherent to how agents approach complex work.

Operational excellence emerges when these orientations are deliberately orchestrated. A system that overweighs process becomes rigid. One that overweighs insight becomes paralyzed by analysis. One that overweighs action becomes chaotic. The craft of operational leadership lies in creating a work operating system where each orientation plays its role without dominating the others.

The Pitfalls That Undermine Operational Excellence

Most endeavors don’t fail because they lack intelligence or effort. They fail because they fall into predictable traps. Some of the most common include:

1. The Illusion of Alignment

Teams often believe they share a common understanding of goals, constraints, and priorities. In reality, each orientation interprets the mission differently. Without explicit alignment rituals, these differences compound into drift, rework, and conflict.

2. Process Overreach

In the name of efficiency, organizations often add layers of process that obscure rather than illuminate the work. This creates a brittle system where compliance replaces judgment and where deviations (often the source of innovation) are treated as defects.

3. Invisible Work and Hidden Queues

When work is not made visible, bottlenecks form silently. People optimize locally, unaware of the system‑level consequences. Operational excellence requires exposing the true shape of work, even when that visibility is uncomfortable.

4. Over‑reliance on Heroics

High performers can temporarily mask systemic flaws, but heroics are not scalable. Systems that depend on exceptional individuals inevitably collapse under load. Excellence comes from designing systems where ordinary performance reliably produces extraordinary outcomes.

5. Fragmented Feedback Loops

When insights from the front lines don’t reach decision‑makers—or when decisions don’t translate into operational behavior—the system loses its ability to learn. Excellence requires tight, bidirectional feedback loops that connect strategy to execution.

6. External perturbations

Changes are inevitable, and the dynamics they generate amplify the effort required to resolve known problems and obfuscate the clarity of purpose essential for coherence.

The Real Work: Integrating Perspectives Into a Coherent System

Operational excellence is ultimately a systems integration problem. It requires building a work operating system where:

• Strategy is translated into actionable constraints

• Processes are designed to support - not constrain - human judgment

• Insights flow freely across boundaries

• Execution is measured in terms of outcomes, not activity

• Agents understand not just their role, but how their orientation contributes to the whole

This is why operational excellence is so difficult: it demands continuous negotiation between competing goods; efficiency vs. adaptability; standardization vs. autonomy; speed vs. quality. The organizations that excel are those that treat these tensions not as problems to eliminate but as dynamics to manage.

The PIANOS model exposes these viewpoints and operates within a macro-level flow depicted in Figure 2:

A narrative of these viewpoints in action

Here lies the Territory occupied by endeavors pursuing excellence. Its aspirations are charted as requirements, the sacred pact made among team members and the endeavor’s stakeholders. The endeavor’s Audience - the sum of all our stakeholders, vests a Navigator role with fiduciary trust, expecting that these captured requirements will stand as the blueprint for endeavors as they unfold. The interactions with this Audience often shape risk profiles, priorities, and cultural dynamics.

Yet in every endeavor, risks lurk - wily shadows that threaten our course and anticipate unforeseen detours. Opposite them, opportunities wait in latent promise, ready to be elaborated by those daring enough to shoulder the inevitable responsibilities. Navigation allocates resources to fuel learning and knowledge capture which provide a foundation for improvements that will stick. Despite this knowledge, the impacts of changes continue to emerge, perturbations that nudge our venture from stillness into motion.

In parallel, Adaptation lays down Structure, carving channels for Alignment. It’s the architect of our resilience. It beckons Orchestration, that deft hand which conveys vision and assures cohesion, ensuring each resource finds its place.

Actions take form, as each team member uses their competency and capabilities to execute their Jobs to Do. These tasks are fed by ceaseless feedback loops. With each iteration, outputs materialize, as system updates are deployed, releases are delivered, and customers are served.

Between each node, flawed intentions and slips also flow contaminating our aspirations and our gaps. Defects emerge where intentions falter, and power pulses to correct our course. The crucible of alignment serves to confirm that business objectives are being effectively translated into operational targets and trigger adjustments when warranted.

Within this Territory of intentions therefore lies both our greatest potential and our most familiar pitfalls. Only by tracing these threads, and by orchestrating Adaptation and Navigation, can we shift from planning to effective action which will transform intentions into impacts.

1. what outcomes are expected from this work?

2. why is this work required?

3. who is going to do and use the work?

4. when must this work be completed?

5. how will the work be performed?

6. where will this work be reviewed and reported on?

The ‘what’ question should be answered by writing down the accomplishments and completion criteria which must be satisfied when performing this work. These criteria should enable an evaluation of whether an event has been completed or not, with the determined result producing a binary choice. While activities may be represented in relative terms (like “50% complete), accomplishments are either achieved or they are not. A good plan needs a description of exactly how this doneness will be determined. A good way to do this is to document the tangible evidence that is expected to demonstrate satisfactory completion, such as the examination or evaluation of an artifact, rather than just tracking the passage of time, or the consumption of resources. Events described in this way provide the basis for assessing progress while the work is being performed.

Answers to the ‘why’ question reveal the underlying intentions for performing the work. If the team’s performance ever strays too far from the path of these intentions, a record of the underlying motives is helpful in providing a north star for reorienting the team as they consider whether to adjust their plan or their execution.

Answers to the ‘who’, ‘when’, and ‘where’ questions are typically required to divide up the work into executable chunks by performing units. For major endeavors, this partitioning is often done by defining distinct work packages that can be separately estimated and planned at lower levels. These work packages are natural subdivisions of lower-level activities that can be assigned to specific organizations for allocating responsibilities, lower-level planning, and monitoring progress during execution.

A work package is completed when all accomplishments assigned to that work package have been satisfied. To answer these three questions, the description of these work packages should record the work that is to be performed, the deadlines by which that work must be completed, and the skills and expertise required for performing that work. Once the work package is framed in this way, an estimate of resources required for this execution. and an assessment of the risks that may impede progress along the way should be developed. For example, work packages might be defined for performing a design task, developing tooling, making an initial manufacturing production run, performing a shop order, purchasing an off-the-shelf component, or any other definable collection of activities which can be assigned to another organization or a sub-unit of an internal organization.

The ‘how’ question is answered by describing the activities which will achieve these outcomes. These activities often require elaboration at different levels of detail over time, though it is unusual for all of the required details to be known before execution must begin. Since planning for complicated endeavors are required at many different levels, the associated plans and schedules are established and managed in a hierarchical fashion. When we use the adjective ’master’ with the noun ‘plan’, we are mean the top-level elements of this collection, which are the principal, driving forces, and controlling authority, for lower-level activities. In this context, the top and upper-level elements will likely be established using different methods, and will be negotiated by different parties, than lower-level plans. Detail is added, and accountability is provided, as the schedule, functional, and performance requirements are flowed down, and managed within a well-defined set of organizational interfaces. When we add the adjective ‘integrated’ to our ‘master plan’, we have an ‘integrated master plan’, which describes the details and information relationships across these multiple levels, and assures that they have been verified to be sufficiently consistent and complete for execution to begin, and refinement of planning to occur over progressively longer time horizons.

While planning can occur in people’s heads, and in conversations among team members, none of that has meaning unless it is written down and approved. An integrated master plan must be written down, reviewed, and approved. This suggests using an outline such as that provided below:

• Introduction

  • A short description of the endeavor

  • A definition of the scope for the work

  • An analysis of the progressive flow and maturity of work products. There should be a particular focus on adopting conventions and expectations for maturity definitions such as draft, preliminary, written, submitted, accepted, and approved.

• Assumptions and Ground Rules

• Events, Accomplishments, and Criteria Dictionary

  • Events, which are the logical points at which to assess progress

  • Accomplishments, which are the desired results to be achieved by a specific event

  • Criteria, which is the evidence required to demonstrate that each accomplishment has been complete

• Work packages

  • Organizational responsibilities for producing the work package

  • The primary organization responsible for negotiating additional details about the interactions between this work package and other work packages (often called the work package owner).

  • A narrative describing the work package activities, with a focus on the handoffs into or out of the work package:

    • what inputs are needed and by when

    • which work package those inputs are needed from

    • what outputs will be produced

    • when (relative the start of the work package) key decisions need to be made

  • The definition of done

The intent of this information is to lay the groundwork for producing a schedule which is tethered to a start date, i.e. weeks from launch, months from go-ahead, etc.

To avoid confusion about whether a particular named activity is located in the plan, or will instead be found in its accompanying schedule, an effective convention for naming is often adopted; this convention involves describing activities in an integrated plan using past tense verbs, since these activities describe assessment points that should have been completed when the event itself occurs. For example a key event might be described using accomplishment named ’Systems Requirements Review completed’. In contrast, present tense verbs should be used to describe activities when they are instead in the integrated master schedule (e.g., Develop Software Specification, Perform Requirements Analysis). This change in verb tense reinforces that there is a relationship between planned activities and scheduled activities and makes it easier to integrate activities and events across plans and schedules. Consistent use of such constructs helps to emphasizes these important distinctions and provides a very simple naming convention that signals where to find things.

If the definition or development of any portion of a work package has not yet been sufficiently defined to allow planning to be done, a mechanism should be adopted which captures the associated activities in a ‘parking lot’ construct. The need for this deliberate delay in detailed planning typically arises when the work is sufficiently far in the future that it is dominated by unknowns, being deliberately performed in a series of implementation phases, or if scouting explorations to discover which rainbow to follow in order to accomplish that work have not yet been accomplished. Ideally, a rough estimate of the time and resources required for accomplishing this future work can also be made, though this many not always be possible. One way of addressing uncertainty in this context is by producing an estimate with both an upper and lower bound combined with a confidence level associated with this prediction.

The collection of these ‘future planning packages’ should be time-phased in accordance with known need dates and historical rates of execution for similar endeavors. Refined planning of these future planning packages can then be accomplished once requirements and the environment become clearer, and the timeframe for performing the work draws nearer. The work required to refine the plans for these future planning packages should be incorporated as a part of the baseline integrated master plan itself, to avoid painting the team into a corner.

In small bites, a schedule depicts activities and their dependencies for an endeavor. But when endeavors are complicated and involve many pieces, it is likely that a particular kind of schedule is used - an integrated master schedule which links the information in an accompanying integrated master plan for an endeavor, instantiates that information through the network logic established by lower-level schedules, and optimizes the allocation of available resources and time. Through this means, the integrated schedule establishes dates when key events are expected to occur, and weaves together the work packages necessary to achieve the corresponding accomplishments.

An integrated master schedule is thus a time-based depiction of the information contained within the integrated master plan, with substantiation of this depiction through lower-level schedules which implement that plan, and depict how the work will be performed. The use of the adjectives ‘integrated’ and ‘master’ in this context have the same meaning as they do in the corresponding integrated master plan.

A good schedule requires elaboration of details and concurrent sizing of the work to be done. Each require collection of facts and data about that work, so that appropriate modeling of this work can be performed. The organization which charters the development of such an integrated master schedule must commit to proactively use the resulting information (instead of optimism, opinions, or ill-defined targets); it must also then manage the lower-level schedules, or the resulting top-level schedule will be meaningless. While many think schedules are a joke, that attitude is a self-fulfilling prophecy.

In reality, schedules are just a tool, and should not be confused with the challenges that may actually be encountered while performing the work. The process of developing a schedule can be as important as the schedule which results from that process, since this journey itself often serves as a catalyst for discussions which may not otherwise have happened. In the absence of sufficient attention to planning and scheduling, precious time and resources may be wasted on relatively unimportant activities as the work unfolds, and many options for mitigating risk will be missed until it is too late to exploit them.

Work packages owners (i.e. those primarily responsible for producing the results expected from the work packages) must work together to translate the activities, events, and accomplishments described in the integrated master plan into specific dates by:

• aligning with the equivalent planning & schedule models of the higher and lower level schedules for an effort

• connecting with upstream, concurrent, and downstream performing organizations

• working through the needs of customers, including those of downstream performing organizations

• negotiating interactions with other work packages owners

• taking account of estimates for the demand and the availability of critical resources

An integrated master schedule should be presented as a structured hierarchy which make the key activities, milestones, and interdependencies between the constituent work packages visible. An integrated master schedule models the logic of these interactions (i.e., their predecessors, successors, and associated lag or float time) in order to depict how accomplishments will be achieved with the passage of time, and through the expenditure of resources. This hierarchy should also align with realistic start dates and needed end dates to lower level planning efforts. When seen from this perspective, an integrated master schedule is really a network of commitments. The combination of these commitments for individual work packages, their predecessor and successor relationships, and their triggering conditions, are what enable a critical path analysis to be performed, and resources to be optimized across a portfolio.

The integrated master schedule should be a living document that is regularly updated to reflect the progress of the program or project. The information in the integrated schedule network should be logically consistent, and the collection of resources required should sum to the budget agreed to for the endeavor. This collection should also:

• Maintain consistency with the integrated master plan

• Illustrate the interrelationships among events, accomplishments, criteria, and tasks

• Maintain consistency with the work package definitions

• Indicate the start and completion dates and duration for each event, accomplishment, criterion and task;

• Provide the ability to sort schedules multiple ways (e.g., by event, by work package owner, by type of funding required,?etc.

• Provide for critical path analysis;

• Provide the means to perform ?what if? schedule exercises

• Provide for updates on a regular basis, and record completed actions, schedule slips, and rescheduled actions from the previous schedule for reference

This network should include all the elements necessary for achieving the accomplishments. For example, if the project in question is an engineering development project, it should include requirements elaboration and clarification, design and development, verification and validation, production or modification, and delivery of the total system. The work required for these activities should be described within a context which is meaningful to running the business. This network should provide the basis for critical path analysis, establish a framework for identification and assessment of disconnects, and allow alternative scenarios to be explored. The progressively increasing level of detail in these schedules across the schedule network or hierarchy should include:

• A summary schedule, typically depicted in a single view, which highlights the period of performance, key milestones and/or deliveries, and other significant events

• Intermediate schedules, which represent work package or project activities, milestones and phases at a level of detail which is below, but vertically traceable to, the summary schedule. Intermediate schedules are used within performing organizations to manage their commitments and communicate their progress

• Detailed schedules at the level that the work is being done within the formal network, and are the sources of the data that drives the summary and intermediate schedules. Detailed schedules elaborate authorized work into a logical sequence of time-phased tasks and interrelationships, and provide meaningful indicators of work being accomplished at levels below the formal reporting threshold. For example, such detailed schedules may not track individual change requests, but instead would the activities which must be implmemented to deliver a collection of these work requests over time.

As this elaboration is performed, the resources required to accomplish these work packages can begin to be rolled up and examined in aggregate. It is at this point that the total demand on available resources will become apparent. Generally, adding resources to meet the demand should be a last resort, because most resources are not interchangeable, and growing the team will increase the costs of coordination, communications, and decision-making. A preferred alternative is for the scheduling process to rearrange the order in which these work packages will be executed; however, this process should not try to change the underlying structure or approach to doing the work, unless the associated integrated plans are themselves updated.

Negotiations typically begin in earnest when such resource or time conflicts become apparent. A strategy will need to be developed so that tradeoffs among priorities and alternatives can be performed. Ideally, these tradeoffs should avoid erosion of contingency buffers which are necessary to absorb the inevitable risks that will arise. To the extent possible, this reconciliation should be performed so as to preserve the ability to accomplish events in the higher-level schedules, though often, the results of these discussions may also rippleinto requested changes to the integrated master plan itself.

For complex projects or portfolios of related projects, use of a work breakdown structure (WBS) will be helpful to organize the work packages and activities associated with an entire portfolio of projects associated with some endeavor. This work breakdown structure provides a framework and context for collecting costs, reporting progress, and providing oversight by different levels of management, and across different levels of this schedule hierarchy, with respect to the actual status of the individual elements contained in this structure. The work breakdown structure may also organize this work across major phases, such as pre-acquisition, acquisition, and sustainment, to emphasize the necessary lifecycle implications of an endeavor over some planning horizon.

Events and deadlines are typically represented on a schedule using a milestone notation. All such notational conventions should be established up front, for consistency across the schedule network. These standards should also describe how the planned start and finish dates of activities, and the relationships between these activities, should be depicted, as well as provide a means of communicating and managing schedule baselines. As work is performed, a similar convention should also be adopted to depict progress which has been achieved. This convention should also draw attention to the impact of prior delays on work in progress, as well as work planned in the future.

The basis of estimates for the durations of activities should be traceable to realistic projections of effort for the size and complexity of all products which must be produced. Whenever possible, the analysis behind these projections should be calibrated with data from prior experience. While an integrated master schedule should be an event-driven depiction which has been derived from the projected durations of the underlying activities, there may be specific constraints on when individual events must be accomplished by (i.e. no later than dates). A periodic schedule analysis should be performed to evaluate the realism of any sequence of activities relative to such constraints, as progress unfolds.

The path originally envisioned by the integrated master schedule may turn out to be unrealistic, once execution begins; those who are responsible for managing to changes to the schedule should keep in mind that the map is not the terrain. High-risk, concurrent activities may be attempted, and aggressive estimates may be prescribed to meet emerging conditions. For an integrated master schedule to be effective, it should not override what can realistically be done; instead, schedule risks should be identified, and timely mitigation efforts should be put in place to attempt to shorten time flows which don’t fit into overly ambitious targets. The integrated master schedule should then explicitly include these known risk reduction activities which are expected to mitigate these risks, and include the associated resources required for performing this mitigation.

Scheduling tools and the specialists who use them can steer projects into trouble if these tools are used to spread resources in artificial ways, such as by capping resource consumption at some level or forcing the spread of resources to match available funding. Handoffs between individuals are rarely clean unless there is a robust process in place which communicates the protocols of these exchanges. For example, while gantt charts can be useful representations, the activities which they depict can be misleading unless they are integrated, and their start and end dates match the criteria established in an integrated master plan. The resulting allocation may not actually be practical or realistic, but those reading the schedule will not be able to tell the difference. People can’t multitask from one task to another without reorientation and loss of efficiency. Another pattern often observed is for teams to locally authorize shifts to events for their own convenience, without requiring corresponding reviews and authorization of the direct and indirect impacts on other work packages.

There are many practical scheduling guides available that offer good advice about how to satisfy the above provisions. One of the better ones is the Planning and scheduling excellence guide, which is available from the NDIA. This guide provides eight over-arching tenets for building, maintaining, and using schedules using what is described as ‘Generally Accepted Scheduling Principles (GASP)’.

“Planning” is a concept with baggage. For many, it calls to mind a passive activity: sitting, thinking, staring into space, abstracting what you’re going to do. In its more institutional form, planning is a bureaucratic exercise in which the planner writes reports, colors maps and charts, programs activities, and fills in boxes on flowcharts. Such plans often look like train schedules, but they’re even less interesting. Much planning does fit that bill.

And that’s a problem because it’s a serious mistake to treat planning as an exercise in abstract, bureaucratic thought and calculation. What sets good planning apart from the rest is something completely different. It is captured by a Latin verb, experiri. Experiri means “to try,” “to test,” or “to prove.” It is the origin of two wonderful words in English: experiment and experience.

Plan-driven methods

Professional societies offer plenty of guidance about this experiential experimentation. Since 1969, the Project Management Institute has championed plan-driven methods that promise to produce a robust project plan. Unfortunately, the robustness often drags along an accompanying bureaucratic overhead with it. Many other disciplines, like systems engineering and software engineering, echo significant elements from PMI’s framework.

Figure 2 captures the major processes in the Project Management Body of Knowledge Guide (PMBOK), with planning processes getting most attention, and plans being the primary output.

There are ten core elements of the planning process: scope planning, scope definition, activity definition, resource planning, activity sequencing, activity duration estimating, cost estimating, schedule development, cost budgeting, and project plan development. In comparison, there is only one executing process and two controlling processes, so the primary output of project management is the project plan, which itself risks becoming bureaucratic. Quoting the PMBOK:

The definition and management of processes is key to plan-driven methods. For that reason, plan-driven methods are almost always associated with process improvement. The processes need to be defined, standardized, and incrementally improved to provide the data needed to control and manage their operation. Such processes generally include detailed plans, activities, workflow, roles and responsibilities, and work product descriptions.

Thus, learning and improvement are expected to be a fundamental feature built into project plans, though these attributes often rely on activities outside the scope of the resulting plan.

In its most recent incarnation, the PMBOK has placed an increased emphasis on principles, which are worthwhile, unsurprising, and easier said than done. This is a positive step away from a strict control orientation (note, for example, the guidance on the project manager chief role responsible for coordinating project activities). Several concerns about the PMBOK remain:

1. Structural rigidity and lack of flexibility

   1. Prescriptive nature: PMBOK has traditionally emphasized rigid processes and documentation, which can hinder adaptability in dynamic or innovative projects.

   2. Slow to adapt: Critics argue that PMBOK struggles to keep pace with rapidly changing business environments, especially in tech and creative sectors.

2. Documentation focus and administrative burden

   1. Heavy emphasis on documentation: While useful for compliance and audits, extensive paperwork can create unnecessary overhead, particularly for small endeavors.

   2. Process-centric bias: The focus on process groups and knowledge areas may lead to bureaucratic inefficiencies in fast-moving projects.

3. Insufficient emphasis on sustainability

   1. Sustainability gaps: While PMBOK mentions sustainability, it lacks actionable frameworks for integrating related concepts into project success criteria.

   2. Lifecycle sustainability: Critics call for sustainability to be embedded throughout all project phases, not just acknowledged in theory.

4. Underutilization of AI and Data-Driven Decision Making

   1. AI as an afterthought: PMBOK has been slow to embrace AI as a strategic partner in project planning, risk assessment, and communication.

   2. Missed opportunities: Tools like generative AI and machine learning could enhance forecasting, stakeholder engagement, and adaptive planning, but are not yet central to the PMBOK framework.

5. Challenges with Distributed Teams and Modern Collaboration

   1. Limited guidance for remote and distributed teams: PMBOK has traditionally focused on co-located teams and hierarchical structures, which may not reflect today’s collaborative, networked environments.

6. Disconnects between conceptual and practical elements

   1. Theory-heavy: Some practitioners find PMBOK too abstract or academic, lacking practical tools for real-world challenges.

   2. Principles vs. execution: The shift to principles-based guidance in newer editions aims to address this, but implementation clarity remains a concern

Plan-driven methods are themselves production processes which are expected to facilitate tradeoffs across alternative approaches for activities, confront tensions between autonomy and control, and transform inputs into outputs within whatever organizing framework has been adopted for activities. The Origins of Efficiency outlines the challenging circumstances of production:

Production processes are sociotechnical systems, embedded in human organizations and subject to the vagaries of human decisions, which are often guided by considerations other than cost. These decisions can and do influence the trajectory of production process development, by adding costs in service of achieving other goals like increasing safety or reducing environmental harms, or by limiting the sorts of improvements that can be made.

Information is an input needed, and an output produced, by most production processes. When we think of planning as itself a production process, the is often operating in the realm of ideas, narratives, and concepts, often (much of it unreliable), assumptions (when information is unavailable), and predictions of forward-looking behaviors (which are subject to biases), all under time pressure that often limits a necessary understanding of the presenting situations of the endeavor.

Agile

Agile software development emerged in 2001 and provides an iterative, incremental, and evolutionary framework which embraces values and principles appeal to small teams. Since then, many separate methods have emerged that share DNA with these ideas, while seeking differentiation across the competitive environment they operate within; an ocean liner of businesses (including many led by Agile Alliance members) emerged ready to offer their services and combat the common pitfalls encountered with these methods. Yet studies on failed transformations highlight preparation issues: - lack of leadership, inadequate sponsorship, unclear decision rights, misaligned incentives, rigid governance structures, ow psychological safety, and poor cross-functional collaboration.

Many Agile-focused methods have also been embraced in applications outside software development, with limited success. Scaling up agile to major development efforts (100+ developers working on a specific product) and in regulated, safety-focused endeavors has also proved challenging.

Steve McConnell’s book, More Effective Agile offers the following contrasts between plan-based, sequential methods like PMBOK and adaptive methods like Agile:

Comparisons of Agile and plan-driven methods often contrasted failed plan-driven projects with successful Agile projects, or vice versa. Since all successful projects rely upon effective organization and oversight, customer collaboration, and robust transformations, such comparisons are rarely useful.

One of my favorite books evaluating improvement frameworks is Balancing Agility and Discipline. It describes the tension between these two competing worldviews:

The agilists rail against the traditionalists and lament the dehumanization of software development by “Taylorian” reductionists who worship process. The establishment has responded with accusations of hacking, poor quality, and having way too much fun in a serious business. True believers on both sides have emerged to proclaim their convictions with near-messianic stridency, raising the perplexity level of software developers and managers trying to evolve their success strategies.

At the ten-year anniversary of Agile’s Manifesto, Philippe Kruchten captured the ‘elephant in the room’ nature of issues which the participants reflecting on. A sample of his thoughts deserve repeating:

While the community has been good at amplifying what works, it often has not been good at dampening what did not work, analyzing why it did not work, or under which conditions it did not work (context!)…

In most instances, when practices are described too little of the actual context is described, leaving an impression of very wide applicability; even if this wide applicability is not actually claimed. Sometimes this is just pure dogmatism, or bigotry…

Unlike other disciplines, like medicine for instance, there is very little reporting of failures or limitations in software, and no clear venue or even templates or exemplars of such reporting. And again the fear that this would reflect badly on the person reporting, or the “victim” or “culprit” organization. A good template for such report would include a detailed description of the context…

There are still too few people gathering objective, significant, scientific evidence on our practices, except for a very few that are easy to instrument (pair programming, for example)…

The agile movement is in some ways a bit like a teenager: very self-conscious, checking constantly its appearance in a mirror, accepting few criticisms, only interested in being with its peers, rejecting en bloc all wisdom from the past, just because it is from the past, adopting fads and new jargon, at times cocky and arrogant.

Continuous flow

The book The Origins of Efficiency highlights one of the primary constraints limiting both plan-driven methods and Agile: they simply do not scale:

When we take a 30,000-foot view at where and how efficiency has improved, one recurring theme is the importance of scale. Economies of scale are not only a major driver of efficiency improvements in and of themselves, but they are also a gating mechanism that unlocks other efficiency improvements by making it possible to amortize the large fixed costs that such improvements often require over a sufficiently large output…

When we can’t repeat a process, or when there’s significant variation in the production environment or in the product itself, many paths toward increased efficiency are blocked. Without repetition, we can’t take advantage of economies of scale. A constantly changing process or a highly variable environment is harder to predict and control, making it more difficult to increase yield and decrease waste. It’s also harder to automate or mechanize in such circumstances, which limits our ability to reduce labor costs. This is one reason why services like medical care, car repair, and construction, which rely on skilled labor that can flexibly adapt to highly variable conditions, remain expensive. Finding ways to make these things cheap will require escaping the constraints of scale to get the benefits of a continuous process—that is, a constant transformation of inputs into a finished product without waiting, buffering, failures, or wasted actions—without needing to repeat that process over and over again…

The development of continuous processes has historically been most easily achieved with a carefully orchestrated arrangement of workers and special-purpose equipment, which is costly to build and operate and, therefore, requires the cost to be spread over a very large output. It takes time, effort, and expense to make the steps in a process work together in sync, a result that can often only be achieved with dedicated equipment and justified with large production volumes…

What if it was quick and easy to figure out the order of steps in a process and arrange them so that the transformation is swift and steady? And what if this process didn’t require special equipment but could use flexible, adaptable machinery able to produce a variety of different items? Then it would be possible to achieve a smooth continuous process that worked in a highly variable environment, bringing the benefits of efficiency improvement to domains that have traditionally resisted it…

For production technology to be able to flexibly adapt to a highly variable environment, there are two broad requirements. First, it requires [methods and] equipment that is physically capable of responding flexibly—that can take a broad range of possible actions, or a small number of actions that can be combined to produce a wide variety of outputs…

The second requirement for flexible, automated production is that the equipment has some method of determining the sequence of the proper actions to take—an information-processing component that can, based on data from the environment, break a desired output into a series of achievable production steps.

The capabilities available from satisficing these requirements will be considerably enhanced as artificial intelligence technologies are incorporated into activities and allow dynamic reallocation of efforts to maximize achievable outcomes.

Operations focus

Neither Agile nor plan-driven methods provide the universal flow-oriented viewpoint for production that is offered by Lean (an extension of the Toyota Production System). Lean and JIT (Just-In-Time) methods emerged from industrial engineering with a queueing-theory backbone, pushing teams toward dynamic flow while retaining and customer focus.

However, Lean lacks the disciplines of plan-oriented methods, since it is designed for improving operations; further, its focus on elimination of waste can shortcut strategic efforts. As a result, Lean-Agile hybrids have been effective in industries that require both operational excellence and rapid innovation. These hybrids typically feature adaptive planning with feedback loops, pull-based scheduling, visual management techniques, and continuous improvement concepts with a customer-value focus across cross-functional teams.

Analysis

Well-established professions typically build their concepts around theoretical foundations. In The underlying theory of project management is obsolete, the authors recognize that production has many manifestations:

… competing theories of production (projects are just special instances of production) have existed even before the emergence of project management. Another concept of production was presented already in the framework of early industrial engineering. The flow view of production … is embodied in JIT and lean production. In a breakthrough book, Hopp and Spearman (1996) show that by means of the queueing theory, various insights, which have been used as heuristics in the framework of JIT, can be mathematically proven.

While Lean concepts rely on this theoretical foundation, and have recently gained traction among startups, they have had their greatest success fighting ‘big company disease’, through a focus on customer concerns and a systems approach across collaborative teams. Their biggest contribution has been helping to change thinking from a static plan-oriented focus to a dynamic flow-oriented model.

Let’s break down the features of each of these frameworks, drawing from this analysis:

If the reader studies this table, they may conclude no single option provides what they need.

Hybrid approaches

Hybrid approaches, while appealing, aren’t a panacea; common pitfalls include overhead and complexity, fragmented communications, dilution of core principles, tooling challenges, inconsistent metrics, and the ever-present resistance to change. Examples cited in the literature include:

• Teams using sprints but still following rigid upfront planning, losing Agile’s adaptability

• Isolated Agile adoption in which Agile is adopted locally but is not supported or compatible with other departments frameworks, creating delivery misalignment

• Organizations cherry-picking practices and adopting Agile rituals (like stand-ups) without embracing its principles, often resulting in limited adaptability

• Lean’s reliance on accurate forecasting clashes with Agile’s responsiveness, especially in volatile markets.

This is made worse when one realizes that few endeavors fully implement all the features in their situation, so relying on a hybrid puts the user in the difficult role of integrating parts designed for separate purposes on their own.

The bottom line:

The idea of a fixed method, or a fixed theory of rationality, rests on too naive a view of man and his social surroundings - Paul Feyerabend, in Against Method

In the next post in this series, we will confront the inevitable tensions between desires for autonomy and control.

Profit Beyond Measure details how two extremely profitable manufacturers, Toyota and the Swedish truck maker Scania, have rejected the traditional mechanistic mindset of managing by results that generates waste. Johnson and Broms explain how Toyota and Scania achieve their legendary cost advantage through a revolutionary concept they call managing by means (MBM). Instead of being driven to meet preconceived accounting targets, the production systems of Toyota and Scania are governed by the three precepts that guide all living systems: self-organization, interdependence, and diversity.

Amid a wealth of new insights into Toyota’s vaunted system, Johnson and Broms introduce the tools of MBM to show how design, production, and profitability analysis are done to customer order. They demonstrate that by following the principles that emulate life systems, even a lean and profitable company can organize work to greatly lessen its long-term earnings instability and sharply reduce its short-run operating costs.

Scania has achieved sixty-five years of financial stability and longevity in the face of fierce competition. Toyota has amassed a market value since 1988 that has rivaled - or sometimes surpassed - the American “Big Three” automakers combined. The principles that Johnson and Brams set forth in Profit Beyond Measure can guarantee the same richer, longer life to any company that applies them.

Just which jobs we consider complicated and which ones we consider elementary, even crude, is a judgment too often made not on the true nature of the work but on the things that attend it — the pay, the title, whether it’s performed in a factory or an office suite, in blue jeans or a gray suit. And while those are often reasonable yardsticks — a constitutional scholar may indeed move in a world of greater complexity than a factory worker — just as often, they’re misdirections, flawed cues that lead us to draw flawed conclusions about occupations we don’t truly understand. We continually ignore the true work people and companies do and are misled, again and again, by the rewards of that work and the nature of the place it is done.

It may not be much of a surprise that bosses and economic theorists don’t fully appreciate the complexity in many jobs. Both groups, after all, are interested mostly in performance, never mind how it’s achieved. More surprising is the fact that coworkers are often just as poorly informed about what the person one desk over or two spots down on the assembly line actually does all day. When they do give the matter any thought at all, they almost always conclude that the other person’s job is far less complicated than it is.

Feedback is essential for any performer to adjust their actions and converge on the output needed:

The longer it takes for the complexity intrinsic in the work to become apparent, the less useful feedback will be in guiding implementation and incorporating changes that will have lasting effects on performance. People are motivated to change their behaviors when their own interests are connected to business needs. Causes must be connected to effects so they can make sense of information and how it relates to a particular context and framing of their work. In The Principles of Product Development Flow, Don Reinertsen describes the role feedback plays in enabling teams to establish control over their environment:

Rapid feedback causes us to subconsciously associate our delivery of a work product and the beginning of the next process step. This attribution of causality gives us a sense of control, instead of making us feel we are the victim of a massive bureaucratic system.

In the book Improving Performance, Brache and Rummler describe the factors which affect the ability to perform work.

Collectively, these factors determine the extent to which planned work can be successfully executed and determine the extent to which the means of performing such work can be improved over time. These factors include:

1. Job Preconditions

   1. Are the inputs accessible and suitable for processing?

   2. Can the task be performed without interference?

   3. Are the procedures and workflow logical?

   4. Is help available if required?

2. Performance specifications

   1. Do job standards exist?

   2. Are they attainable?

   3. Are they accepted by the people who must perform the work?

3. Ability to execute

   1. Are the roles that individuals are assigned to perform consistent with their experience and preparation?

   2. Do the performers have sufficient capacity available to implement these roles within the time allocated?

4. Feedback

   1. Will feedback information be relevant, accurate, specific, understandable, and timely?

   2. Will the feedback be provided in a way that is useful to the performer and so that adjustments can be incorporated within the required period of performance?

5. Motivation

   1. Have incentives and consequences been established to support desired performance?

   2. Are the criteria for reporting progress and estimates of remaining work well designed, agreed to, and appropriate to the situation?

Simply put, the degree to which work is actionable determines the pace at which work can be acted on, and the extent to which rework can be avoided. Jobs are 'actionable' if and only if they satisfy all the above attributes. When some portion of a work package does not satisfy these attributes, the time it will take to perform that work will be more variable, since work products that will be produced will be more likely to have less utility than desired, and risks associated with achieving target outcomes will be increased. Teams select the most actionable work from a work package to perform first, even though this work may not be the most important work from the perspective of the customer; while it may enhance the team's ability to demonstrate progress in the short term, this approach may not produce the best long-term results.

Purpose

A Concepts of Operations artifact (aka "ConOps") should provide a useful overall depiction of needed capabilities using compelling visual representations. The information should provide a user-centric bridge from the operational goals and objectives of the system of interest to an operational view of how the system will be used within the environments targeted for deployment.

The narrative developed in the ConOps should limit the problem space for defined audiences over distinct time periods. It should document the sponsor and customer assumptions and intentions with respect to each operation or series of operations, especially when a series of connected operations must be carried out simultaneously or in succession. The resulting narrative perspective weaves the associated concepts for a system into a story about how the anticipated physical and technical capabilities of the system will be used to provide selected capabilities, such as resource or information aggregation, computation, synthesis, navigation, production, distribution, or monitoring over time.

A ConOps should provide powerful graphics that reinforce a written narrative that describes a 'big picture' perspective to the designers of the system. Its depiction should amplify and clarify the purpose of the endeavor. This perspective (often composed of multiple viewpoints) should help explain how a proposed solution is expected to work 'holistically' once it enters service. The interactions between solution elements should be collectively introduced in a way that captures the voice of the customers and emphasizes how the delivered capability will enable the harvesting of value by its stakeholders. Use case diagrams can supplement a ConOps further and frame the problem at a lower level but should not be necessary to characterize the essential attributes of the proposed solutions.

Expected Outcomes

A ConOps puts these interactions into a meaningful framework by describing:

• What outcomes are intended from using the system?

• What novel concepts are crucial to understanding how the system will be used to realize these outcomes?

• How will these concepts be utilized throughout the system's lifecycle?

What resources will users need to operate and maintain the system?

Content description

The following outline provides a suggested template for effectively communicating such information. Content described will vary depending upon whether for a new system of interest that is being developed from scratch, when a major upgrade to an existing system is to be performed, or whether the system is intended to replace an existing system.

• Current System or Situation

• Background, Objectives, and Scope

• Assumptions and constraints

• Operational policies and constraints

• Description of the current system or situation

• Stakeholders and personnel affected by change

• Justification for and nature of changes

  • Reason changes are necessary

  • Description of needed changes

  • Priorities of changes

  • Changes considered but not included

• Description new and changed system elements

  • Block diagram

  • Description of critical functions

  • Description of primary interfaces

  • Key timeline and interactions

  • Measures of effectiveness

  • Operational scenarios & modes

    • Initialization

    • Operations

    • Support

    • Modification

    • Service restoration after failures

• Summary of capabilities provided

  • System execution

  • Data management

  • Operations management

  • Systems management

• Analysis of the proposed system

  • Summary of benefits

  • Summary of limitations

  • Summary of alternatives and tradeoffs considered

Planned capabilities need to be oriented within the missions of each affected organization, with particular focus on:

• A delineation of the areas of responsibility, especially at the boundaries

• A timeline which details time budgets for activities

• A description of how progress will be tracked with respect to performance targets

• the resources allocated to perform each aspect of this mission

• how these resources scale under different scenarios, especially:

  • Innovation: the ability to introduce new materials, technologies, and methods, as well as to enhance traditional work in new ways

  • Robustness: the ability to maintain effectiveness across a range of tasks, situations, and conditions, including pivots in the face of unusual (and potentially unanticipated) states. These should include:

    • Resiliency: the ability to recover from or adjust to misfortune, damage, or a destabilizing perturbation in the environment

    • Responsiveness: the ability to react to changes to the environment in a timely manner

Flexibility should be provided to enable actors to employ multiple pathways to success and the capacity to move seamlessly between them. This contingent flexibility should include the following:

• The ability to make sense of the situation in which their role is to be performed

• The means for coordination with other actors, to enable them to work to together efficiently within the operational environment

• Knowledge, skills, and experience sufficient to recognize and respond to each situation

• The ability to select and execute a sequence of steps that will achieve the intended outcomes

While this information does not all need to be covered within the CONOPs artifact itself, it will enable the organizations involved to evaluate the clarity of the CONOPS, and its readiness to guide the subsequent efforts transforming the CONOPS information into a working system.

Summary

This ConOps information will be useful regardless of the domain or disciplines which will be involved in its implementation. If more projects used a ConOps approach, their probability of success will be increased, since the follow-on requirements elicitation process will have a better focus, allow clear delineation of responsibilities, and will be reinforced by a description of what “done” looks like. It is also particularly useful in onboarding new personnel to the endeavor.

For more information about using a CONOPS, see MITRE's guidance on this subject.

In Federalist paper #51, James Madison writes:

In framing a government which is to be administered by men over men, the great difficulty lies in this: you must first enable the government to control the governed; and in the next place oblige it to control itself. A dependence on the people is, no doubt, the primary control on the government; but experience has taught mankind the necessity of auxiliary precautions.

In The Square and the Tower, Niall Ferguson highlights further the weaknesses of people in these situations:

…the defect of autocracy is obvious, too. No individual, no matter how talented, has the capacity to contend with all the challenges of imperial governance, and almost none is able to resist the corrupting temptations of absolute power.

Sometimes failures arise as a result of defects injected by agents themselves. But too often, it is a failure to understand the weaknesses of the methods they use and the dynamics of the collective games that are being played while using them. Let’s consider how frequent these problems arise.

A sampling of failures

The author of the book How Big Things Get Done, Bent Flyvbjerg (a professor at Aalborg University in Denmark) has been tracking over 16000 projects across 20+ different fields, and across 136 countries for over twenty years. His findings:

Only 8.5 percent of projects hit the mark on both cost and time. And a minuscule 0.5 percent nail cost, time, and benefits. Or to put that another way, 91.5 percent of projects go over budget, over schedule, or both. And 99.5 percent of projects go over budget, over schedule, under benefits, or some combination of these. Doing what you said you would do should be routine, or at least common. But it almost never happens.

Steve McConnell’s book More Effective Agile records his consulting business’s experience in over one thousand company engagements with Agile endeavors. As you can see, the performing organizations - on average - failed to apply Agile key practices consistently and effectively across this landscape, even when blessed with the benefits of experience from experts. Unless there is a ‘closed loop’ approach to such practices, this is exactly what one might expect given the inevitable cultural resistance that accompanies introducing such practices into routine practice. And without buy-in and practice, everyone hunkers down until another batch of promising silver bullets attracts attention yet again.

Several large endeavors are particularly noteworthy across the landscape of my experience. Such endeavors tend to be ambitious projects that fear delivering reduced capabilities which can erode trust and accountability in their sponsors, providers, and methods. When ‘selling’ the endeavor, the underlying mindset is to offer optimistic projections drawn from limited surveys while downplaying risks and uncertainty. Their common maladies typically involve overreach and minimal learning:

1. Transportation

   1. Puget Sound’s mass transit project, Sound Transit - according to Bloomberg - was to be a showcase that would show that providing better service would attract more riders and generate a virtuous cycle of growth. The current endeavor was approved by 54 percent of voters in 2016 and was to include sixty-two miles of light rail throughout the region, to be completed by 2041; the delivery was phased, and construction was well underway when the Bloomberg article was written in 2018. It has not gone well.
      
      One of the key segments was across the I-90 bridge, which moves with wind, water, and traffic. In 2022, it was discovered much of the bridge installation required extensive demolition and rebuilding (much as other segments have encountered, suffering of all things rail gauge incompatibilities). These issues pushed the original 2023 opening of cross-bridge traffic to 2026. The original $55B budget, which voters (paying over $1700 per household per year in new taxes) were assured had more than adequate contingency for any uncertainty, is now looking at between a $30-40B overrun. Five other segments to suburbs have also suffered delays of up to seven years, while their taxes, and additional overruns, continue. Yet the current system carries less than 1% of all trips in the region, and has failed to offload overcrowded roads, as promised.
      
      Such problems are always viewed as financial, with the only solutions offered by advocates involving additional revenue and deferring implementation still further. Meanwhile, revenues from riders have tanked, as the adopted hub-and-spoke design reduces rider convenience and its required transfers lengthen end-to-end times. Luckily, self-driving cars are beginning to get approval for ride sharing in limited areas, and by the time the Sound Transit plans are completed, this more effective and less costly alternative will likely be ubiquitous.

   2. But it’s hardly the worst transportation endeavor currently in work. Consider California’s High-speed rail effort, authorized by a 2008 statewide initiative to deliver continuous service between San Francisco and Los Angeles with a nonstop travel time of one hundred sixty minutes. Phase 1 (five hundred miles long) was to be delivered by 2020, at a cost of $33B. Phase 1 is expected to share tracks with conventional trains, operating at a higher average rate than France’s fastest TGV trains. They were warned this was infeasible back in 2008. Costs to date have tripled and projected completion is thirteen years after original commitments. It’s no wonder that funding is questionable since the initial construction package will only connect Fresno and Bakersfield, over which little traffic is expected.

   3. How’s NASA doing in transporting astronauts to outer space? We are no longer hitching rides on Russia launch capability, but our astronauts still need a place to ride. The Orion Capsule is NASA’s crew spacecraft capable of supporting a crew of four beyond low earth orbit and has been under development since 2006. Its specifications call for twenty-one days of undocked service and six months of docked service, and are to be used by the Artemis program, NASA’s current lunar exploration endeavor. Orion has already spent $31B (2025 dollars) and struggles to recreate the mission first accomplished over sixty years ago (the total expenditures including the Space Launch System approach $100B). The maladies of the doomed effort are well chronicled here; suffice to say that if only cost and schedule were the problem, one could still kindle some hope, but it’s far, far worse than that. Meanwhile, SpaceX’s private space efforts continue to astound, with a vision pointing to establish a permanent Mars settlement, rather than completing another trip to the moon.

2. Government services

   1. Health care

      The U.S. health care system is innovative but fiscally unsustainable. It delivers cutting-edge treatments and short wait times for specialized care, but at the cost of uneven access and worse population health outcomes than other wealthy nations. For example, according to The Origins of Efficiency, the cost to develop a new drug has increased by a factor of 14 - in inflation adjusted terms - over the last 4 decades. These problems have worsened since the launch of the Affordable Care Act, which has relied on escalating infusions of subsidies, rather than effective cost control.

   2. Education

      Since the founding of the Department of Education

      1. $3 trillion spent

      2. Reading scores for 13-year-olds have not improved

      3. Only 27% of 8th graders proficient in math

      4. Only 31% of 8th graders are proficient in reading

      5. 47 cents of every federal education dollar spent on compliance

      
      If you’re defending the Department of Education, you’re defending a failure. For example, five years ago, according to a recent report from UC San Diego faculty, only about 30 students arrived with math skills below high-school level. Now, that number is now more than 900, with most failing to fully meet middle-school math standards. As a result, the students supported by these investments are expected to earn 8% less than graduates 12 years ago.

   3. Megaprojects

      Look no further than The Line and its collapse.

Traditional frameworks appear to have had inadequate influence on outcomes for many types of endeavors.

Reality always requires investments in understanding

Existing frameworks focus on prescriptive content definitions for expected artifacts (many of which are rarely read or maintained in practice). When an endeavor is large, complex, and ‘too big to fail’, political will inevitably contaminates prudent judgement, a recipe for failure rather than success. A library of excuses - environmental reviews, inflationary costs, adjustments in scope, externalities - is quickly designated the cause, without confronting the uncertainty inherent in snapshots of reality against plans. Plans have at their core imperfect forecasts of the future based upon limited knowledge and less understanding, so assumptions must be made. These assumptions can inevitably unravel in long-lived projects, those guilty of overreach may have moved on to their next adventures, and a new wave of leaders emerge to claim that a comprehensive reset (with new assumptions) will curtail further sins - if stakeholders will just wait a while longer and shell out a lot more. Until they won’t.

True understanding is achieved by through adequate preparation in the ways things can go wrong, as described in the next post in this series.

The tension

Miranda’s musical celebrates Hamilton’s ambition and legacy, framing him as a hero of order and national coherence by favoring centralized, expert-led authority to tackle national problems like infrastructure, regulation, and technocratic governance. This pursuit often occurs in an atmosphere of Jeffersonian skepticism which champions local autonomy and individual liberty and is wary of elite control and centralized power. It’s a romanticized origin story of American governance, emphasizing the creative tension that forged the Constitution and early institutions.

The musical is an entertaining story, but reality is, as usual, more complicated than that. Marc J. Dunkelman’s Why Nothing Works explores why America, once capable of building massive infrastructure and social programs, now struggles to “get things done.” Everyone needs power but would prefer someone else to shoulder the burden of making that happen (Figure 1).

A case study

Dunkelman argues that endeavors often try to embody both of these impulses, which can lead to incompatible goals and conflicts. These conflicts can produce a vetocracy, where too many actors can veto action, stall progress and erode public trust. It creates a dynamic in which no one has the power or motivation to say “yes”, while anyone can delay and block action. The inevitable outcomes of these forces lead to repeated delays that can accumulate and result in cancellations.

This dynamic is highlighted in Chapter 8 of Dunkelman’s book, which uses electrical power distribution as a metaphor for the broader dysfunction in American governance. It illustrates how the tension between centralized control and decentralized autonomy undermines systemic reliability and reform. It also highlights the challenge of big ideas - for example, demanding national action on climate change (a Hamiltonian tendency) while resisting higher-level mandates that impinge on personal rights (the Jeffersonian approach).

In order for alternative energy sources to be developed, the means to transport that power from source to destination is crucial. The U.S. power grid evolved from a patchwork of local systems into a vast, interconnected infrastructure. Initially, power was generated and distributed locally, reflecting Jeffersonian ideals of community autonomy and responsiveness. But as demand grew and industrialization accelerated, the need for centralized coordination became unavoidable. High-voltage transmission networks and regional interconnects were introduced - hallmarks of Hamiltonian planning - and enabled bulk power delivery across states, improving efficiency and reliability, and striving for balance between the benefits of competition and the stability of monopolies that are core characteristics of public utilities.

However, Dunkelman argues that this centralization of planning and control came at a cost, as local actors lost control of their property and resources, and the system became vulnerable to fragmentation and political gridlock. Regulatory agencies like FERC and NERC were created to oversee reliability and standards for these assets, but their authority is often undermined by state-level resistance, utility lobbying, environmental causes, and fragmented jurisdiction. The result is a paradox: while the grid requires Hamiltonian coordination to function, Jeffersonian impulses - manifested in local vetoes, fragmented oversight, and resistance to federal mandates -prevent necessary upgrades, especially in integration and resilience planning for renewables.

This tension is especially evident in ongoing efforts to modernize the grid. For example, building interstate transmission lines for wind and solar power requires federal coordination, but local communities often block projects due to land use concerns or political ideology. Since these sources of clean power require these transmission lines to transport power from producer to consumer, the dynamic erodes action on climate change while resisting the centralized authority needed to implement it. This contradiction mirrors the broader theme of the book: America’s inability to reconcile its founding impulses now paralyzes its capacity to solve big problems.

Progress, far from consisting in change, depends on retentiveness. When change is absolute there remains no being to improve and no direction is set for possible improvement: and when experience is not retained, as among savages, infancy is perpetual. Those who cannot remember the past are condemned to repeat it. In the first stage of life the mind is frivolous and easily distracted, it misses progress by failing in consecutiveness and persistence. This is the condition of children and barbarians, in which instinct has learned nothing from experience. - George Santayana, The Life of Reason

Santayana highlights the importance of focus, experience, and learning over instinct. Just as Hamilton and Jefferson clashed over the role of government, modern America struggles to balance expert-led coordination with grassroots autonomy. Until this tension is acknowledged and addressed, Dunkelman warns, the nation will continue to suffer from systemic breakdowns—not just in electricity, but in governance itself.

A scorecard for traditional frameworks

There are plenty of competing frameworks that presume a control orientation, but all have limitations, and none provide explicit guidance for all the forces captured in the PIANOS model. It is worth stepping back and putting our system’s thinking hat on to understand why.

In their article Applying Systems Thinking to Engineering and Design, Monat and Gannon highlight the key principles of system thinking for production:

1. Holistic Perspective/Proper Definition of System Boundaries

2. Awareness that Events and Patterns are caused by Underlying Structures and Forces

3. Systemic Root Cause Analysis

4. Sensitivity to Emergent Properties

5. Sensitivity to Unintended Consequences

6. Dynamic Modeling

7. Focus on Relationships

8. Sensitivity to Feedback

Many endeavors employ these principles in practice, but none of the traditional frameworks explicitly seek to leverage these principles.

The dynamics

Let’s consider how dynamics of traditional frameworks unfold using the causal loop diagram shown in Figure 2. Such diagrams highlight the interactions of factors involved in achieving some result, in this case, the capacity of a production system. These interactions (labeled by the red arcs) have a polarity associated with each relationship, labeled with a plus or minus sign. This polarity indicates the influence one factor has on another - positive means ‘moves in the same direction’ and negative means ‘moves in the opposite direction. For example, the leftmost arc connects the cost of capital to the economy with a minus sign, meaning that in general (without considering other factors), when the cost of capital increases, the net effect will cause the economy to deteriorate. Alternatively, when the cost of capital decreases, the net effect will enhance the economy.

These factors also collectively form loops which accumulate the impacts of this polarity, and feed back to influence subsequent performance, leading to further growth or decline. These loops are labeled in the diagram with a prefix of R as reinforcing (creating growth) or B for balancing (leading to decline).

When building a simulation of these dynamics, the extent of each factor’s influence can be calibrated to match what is common knowledge, as long as the natural variation that accompanies each factor is taken account of.

The figure highlights the effects of resource contention - resources can be anything with enduring value, such as a home, a person’s health, and even their education. The ‘centralization of control’ factor in the lower left-hand corner is examined in more detail here. The running narrative for these interactions - applicable to any production system (whether governmental or industrial) - follows:

Loop R1: The Capability Investment Flywheel

At the heart of the diagram is a reinforcing feedback loop: production capacity increases → enabling more investments → which fund capability enhancements and learning opportunities → boosting labor productivity → which further expands production capacity. This loop reflects a virtuous cycle of growth, where reinvestment in human and technical capital drives compounding returns. It’s a Hamiltonian vision of centralized scaling through expertise and infrastructure.

However, when the economy takes a downturn - whether from a national crisis or demographic decline - this loop runs in reverse with limited options for recovery, all relying on the ability to attract buyers willing to finance additional investments with higher risk and lower returns.

Loop B2: Human Capital Foundations

This loop highlights how willing and able workers depend on upstream factors like:

• Effectiveness of education

• Effectiveness of public health

• Immigration (as a labor supply enhancer)

These elements feed into labor productivity and ultimately production capacity. But they’re also shaped by regulations, tax burden, and centralization of control, which can either enable or constrain public investment. This loop shows how social systems and governance deeply affect economic output, underscoring the Jeffersonian tension between local autonomy and centralized policy.

Loop B3: Resource Competition Dynamics

Here we see a more complex, potentially destabilizing loop:

• Rising production capacity boosts the economy, which increases resource demands → driving up prices.

• Higher prices may stimulate resource supply, but also increase debt and inflation, which can negatively impact cost of capital and reduce investment.

This loop introduces negative feedback which can increase systemic fragility. It shows how economic growth can trigger affordability crises and financial instability, especially if regulatory and monetary systems fail to adapt.

Loop B4: Resource Supply and Regulatory Friction

Demand on production increases pressure on the supply of resources, which can be constrained by regulations and tax burden.

• If these constraints are too tight, they reduce investment and slow the capability loop.

• If too loose, they risk environmental degradation or inequality.

This loop captures the dilemma this poses for governance: balancing growth with sustainability and equity. It’s where Hamiltonian efficiency meets Jeffersonian caution.

Loop B5: Popping economic bubbles

Once irrational exuberance begins fueling a speculative bubble, the threat of future price increases begins to spur investor enthusiasm, causing a contraction in the economy. There will be winners and losers around this transition.

The dynamic tensions in systems

The diagram doesn’t offer a single solution—it models a dynamic equilibrium shaped by competing feedback loops.

• Positive loops drive growth and innovation.

• Negative loops introduce checks, risks, and social costs.

• Governance levers (regulation, centralization, taxation) mediate these forces but are themselves contested and subject to dynamics.

The lower right factor Consensus on policies deserves elaboration. Consensus - the collective agreement or unified opinion reached by a group - doesn’t require unanimity, but it implies broad support or alignment. That only happens when leadership can orchestrate stakeholder negotiations and interests until a broadly acceptable solution emerges that everyone can live with. That relies upon a common understanding, acceptance, and support of the required tradeoffs which are necessary to realize the goal being pursued - all infused with patience for the dynamics to produce those results.

Achieving balance

In the Incremental Commitment Spiral Model, Barry Boehm emphasizes what he describes as The Fundamental System Success Theorem, which states: “A system will be successful if and only if it makes winners of its success-critical stakeholders.”:

One implication of the Fundamental System Success Theorem is that it is risky to use terms such as “optimize,” “minimize,” and “maximize” in project guidance. As Nobel Prize winner Herbert Simon showed in his book Models of Man, success in multiple-stakeholder situations is achieved not by optimizing, minimizing, or maximizing of individual criteria, but rather by satisficing with respect to the stakeholders’ multiple criteria.

To be accountable for a commitment, it is important to neither promise nor expect more than can be delivered. In a world of rapidly changing mission priorities, new technologies, competitive challenges, and organizational relationships, total commitment to the full details of a product to be delivered five years later is a very risky bet.

The term “satisficing” here means not everybody gets everything they want, but everybody gets an outcome that they can be satisfied with. Lacking such consensus, convergence on solutions may be fleeting, disappointing, and potentially overridden as power shifts with improved situational awareness, emerging political alliances, and new externalities drive dynamics. Adaptation is key to weathering this storm.

In the next post of this series, the scaffolding underneath these tensions is explored to lay the groundwork for understanding the common failure modes in endeavors. With these insights, you will be ready to practice how to avoid them.

The ideas behind the model are not intuitive, and the handbook lays out a strategy to introduce its ideas within an operational context. The graphics have been updated considerably from his prior works, which introduce the model's ideas but may not reveal its nuances.

Businesses have traditionally adopted a top-down command structure and flowed down strategic plans with a cascade of instructions through the ranks; unfortunately, many of these efforts lack effective feedback or the means to exploit it when available. This structure is simply too slow and inflexible to deal with environments presenting increasing rates of change and levels of complexity. It turns out that reporting relationships matter less than understanding how the complex parts of a business fit together to form a synergistic whole.

Yet when more flexible structures are adopted to better enable autonomy, this same freedom may reduce cohesion, meaningful cooperation, and synergy, which constrain the sharing of knowledge and information necessary to respond to situations rapidly and effectively across organizational boundaries. It is not possible to share or digest enough data; leaders can only recognize patterns sufficiently within their locus of control.

In their book, Profit Beyond Measure, authors H. Thomas Johnson and Anders Broms describe how blunt our tools are when we attempt to translate goals into the actions to be taken by others:

Probably the ultimate root of the modern habit of using quantitative abstractions to represent, or stand in place of, concrete things is language, "the mode of behavioral coordination through which humans bring forth the world they create with each other thing from its context". By using words, human beings give abstract shape to the world we occupy. Language lets human being separate a particular thing from its context. Formulated in a phrase, a tangible thing can then be analyzed intellectually, as if it existed apart from its physical context in the real world. Because of language, ?there is no limit to what we can describe, imagine, and relate [as mental abstractions]".

Since language allows humans to objectify and depersonalize particular things, its power is formidable. When we use language to create abstractions and to generalize, we remove and separate particular things from their context and treat them as if they were not part of a pattern in a living system. Thus, the first use of a simple word as "tree" begins to separate us from the natural habitat we share with other living beings. Once we objectify "tree? with a word we stop living among the lungs of the earth in the wild, and begin to see trees as the source of results such as fruit, shelter, tools, transportation, shade, decoration, wind barrier, writing surfaces and toilet paper. By the twentieth century, trees, almost extinct as species, became a harvestable crop in tree farms, where the primary result is to maximize the projection of "fiber" for industrial and management purposes. Thus, managing by result - managing to secure a "sustainable" supply of fiber- subject to the the web of relationships in the natural forest system to the stresses and strains of clear cutting, thereby threatening the long term viability of the natural system.

In the same way that language lets us think abstractly about fiber and trees, language also allows us to treat the abstract concept of the business result as if it were an actual object that existed separately from the messy, organizational context which gives rise to it. Viewed as an independent entity, this abstraction, this quantifiable business result that we entertain only in our minds, seems to be more real and concrete than does the real situation from which it emanated in the first place.

It is not a great leap, then, from seeing abstract results as concrete realities, to trying to manage them by arbitrarily manipulating the relationships from which they emanate. Such manipulation involves separating ends from means, or goals from the acts that achieve them. In Western culture the separation of means (the way our goal was achieved) from ends (the goal or what is to be achieved) has often been accompanied by a belief that the end or goal is immutable and prominent, and the means are ephemeral and changing. Therefore ends seem more "real" and more "valuable" than the means.

The underlying elements described by Beer are collectively holistic and autonomic. Small businesses may exhibit these functions with direct reporting relationships with the owner, whereas in large organizations they may require examination of organization charts with titles and responsibilities to discern their interrelationships.

Key concepts

Variety

In cybernetics, the number of distinguishable items (or distinguishable states of some item) is called the “variety’. So we may sum up by saying that the output variety must (at least) match the input variety for the system as a whole and for the input arrangement and the output arrangement considered separately. This is a vitally important application of Ashby’s Law of Requisite Variety, which says that control can be obtained only if the variety of the controller (and in this case of all the parts of the controller) is at least as great as the variety of the situation to be controlled. This, like all profound statements of natural law, is perfectly obvious once it has been pointed out.

There is no great difficulty, however, in finding examples of attempted control systems which disobey this law quite flagrantly, and therefore do not succeed. From traffic control to the control of the national economy, this fallacy is apparent; indeed, this is one of the key problems of control in a firm. For management always hopes to devise systems that are simple and cheap, but often ends up by spending vast sums of money to inject requisite variety which should have been designed into the system in the first place.

Multinodes

We claim we know how the whole thing works. The problem is to make it work more quickly. That must surely mean the introduction of discipline and order, of some sort, into the situation. It also means, however, that no measures may be adopted which would at the same time put the remarkable freedom of action and the wonderful flexibility of the multinode in jeopardy. If people could see how to do this, without putting themselves and their organization into a straight-jacket, there is some chance that they would adopt new techniques. One method, we ought now to agree, must be excluded although it is the one method most usually attempted in practice, because no one can think of anything else. This is the method of rigorous protocol. Explicitly: it denatures the system itself - with all its in-built capacity to generate the right answers.

The first difficulty is to know what kind of problem the multinode actually solves. It does not devote itself, its seniority and power, to the determination of trivial outcomes or it ought not to do so. It is likely to be settling a policy of great importance and therefore considerable complexity. Thus it is that people think of thinking as a process of synthesizing an integral but elaborate conclusion from a large number of component parts. The decision is seen as a rococo edifice built up clause by legal clause. This is perhaps why there are endless drafting problems facing anyone trying to promulgate an agreed decision.

The cybernetician adopts the contrary position. The output of the thinking process, the decision, has the following form: do this (rather than anything else). When the process of thinking originally starts, the multinode is faced not indeed with a number of building bricks in an edifice but with a seemingly infinite number of possible outcomes between which it must choose. It is the existence of this plethora of possibilities that cries out for decision in the first place. Then, under this model, the process we seek to assist is one of chopping down ambiguities and uncertainties until we may say: do this. In short, we would like to measure the variety of the complex decision at the start, measure the reduction of variety brought about by each conclusion reached in the process of thinking things out, and in general monitor the entire operation of the multinode as the variety comes down to a value of the decision itself. To do this we shall need two tools: a paradigm of logical search, and an actual metric - a rule and a scale - for measuring uncertainty.

Viability

An organization is viable if it can survive in a particular sort of environment.

When people refer to the firm, or any other institution, as ‘viable’, they are often referring to economic viability. From this preoccupation with the economic dimension stems the assumption that most of our problems are economic too.

Solvency, it is true, is a prerequisite of business activity- to trade while insolvent is illegal. Profitability, too, is a prerequisite- lack of business confidence is not illegal, but it is lethal. So these affairs are of primary concern. But they do not (as many suppose) constitute the goals of the enterprise. Rather are they the constraints under which it operates. So of course the regulation of cash-flow (for example) is an important managerial concern. And of course profit-consciousness in the private sector, or the consciousness of social benefit in the public sector, has to apply to expenditure; and proper returns on capital are required.

However: the tendency in ‘the city’, and among financial journalists, to treat all this as the essence of viability, is to mistake the epiphenomena of the system - the appearances and flurries of activity that prove it is actually there- for the system itself. Which of the firm’s workers, or even middle-managers, could recognize the old place by its indices of ROI, P/E ratios, and the rest? These things are abstractions, and very useful ones too, if we want to manipulate successfully our economic constraints.

Requisite variety

A measure of the complexity with which management has to deal.

You may well say that the number of possible states in a complicated entrepreneurial system is not precisely countable. That is surely correct. However, it is countable in principle: it is therefore amenable to the making of comparative statements (this has more variety than that), and to the arithmetic of ordinal numbers (this product is the fifth most profitable).

Resource bargains

The Resource Bargain is the ‘deal’ by which some degree of autonomy is agreed between the Senior Management and its junior counterparts. The bargain declares: out of all the activities that System One elements might undertake, THESE will be tackled (and not those), and the resources negotiated to these ends will be provided.

Principles of organization

1. Managerial, operational, and environmental varieties diffusing through an institutional system tend to equate; they should be designed to do so with minimal damage to the people and to cost

2. The four directional channels carrying information between the management unit, the operation, and the environment must each have a higher capacity to transmit a given amount of information relevant to variety; selection in a given time than the originating subsystem has to generate it in that time.

3. Wherever the information carried on a channel capable of distinguishing a given variety crosses a boundary, it undergoes transduction; the variety of the transducer must be at least equivalent to the variety of the channel.

4. The operation of the first three principles must be cyclically maintained through time without hiatus or lags

Two aphorisms

• It is not necessary to understand the workings of a black box to understand the nature of the function it performs.

• It is not necessary to understand the workings of a black box to calculate the variety that it potentially may generate.

Axioms

1. The sum of horizontal variety disposed by n operational elements (systems one) equals the sum of the vertical variety disposed by the six vertical components of corporate cohesion. (The six are from Environment, System Three*, the System Ones, System Two, System Three and Algedonic alerts.)

2. The variety disposed by System Three resulting from the operation of the First Axiom equals the variety disposed by System Four.

3. The variety disposed by System Five equals the residual variety generated by the operation of the Second Axiom.

System types

Using the architecture of the human cortex as a pattern, diverse range of specialization structures and backgrounds, and weave them into an ability to mitigate the constraints of bounded rationality, and getting reliable results out of unreliable components, especially under conditions of high uncertainty and complexity, by provide redundant inputs and assuring timely closure in decision-making processes.

• What is the purpose?

• What is the direction?

The types are numbered, with lower level (higher numbered) systems using algedonic signals to ‘wake up’ higher systems. Investments are used as variety attenuators.

Multiple 3-2-1 systems operate as peers (and on a traditional organization, would be depicted horizontally), providing separate functions, such as engineering, marketing, production,

Performance components

System 1 - Processes

The primary purpose of system 1 is to sense and make sense of what has been sensed in order to do something that will be rewarded by its environment. Operations are managed through a regulatory center, which fans out its inputs to a variety of measuring points, which are incapable of measuring everything. The overall processes must establish:

• What activities are to be done?

• What resources are to be allocated to perform these activities?

• Who will be held accountable if resource bargains are broken?

• How will operational exceptions be managed, whether to triage issues or adjust behaviors?

• How will business opportunities be sanctioned and managed?

System 2 - Coordination & Orchestration

System 1 activities may get in each other’s way. This can cause oscillations that result from unconstrained schedule glitches, budget disruptions and problems with enabling infrastructure. Operational units should comply with the various policies and standards, not because they are ordered to do so, but because they offer tangible benefits/control/cohesion, resource management, performance management, compliance management, monitoring/audit, or coordination.

Iterations clarify:

• How is coordination or orchestration provided appropriate to the situation?

• What protocols must each multinode perform to support the higher regulatory authority?

System 3 - Internal monitoring and control

This requires self-organization and autonomic regulation, making inside and now decisions across the collection of System 1 activities within their emerging situation, allocating time, money, space and attention. It also must clean up excess variety that that falls through the cracks of regular arrangements.

If things go wrong and levels of risk increase the System 3 asks for help or puts it to colleagues for a remedy. This is the pain of an algedonic alert, which can be automatic when performance fails to achieve capability targets. The autonomic 3?2?1 homeostatic loop's problem is absorbed for a solution within the autonomy of its metasystem.

Measurements are used to match people, machines, and money to jobs that produce useful products or services. In a set of processes, some jobs are done by one person. Some are done by many and often many processes are done by the same person. Throughout the working day a participant, in completing a task, may find the focus shifts between internal and external Systems 1?5 from moment to moment.

The choices, or decisions discriminated against, and their cost (or effort) defines the variety and hence resources needed for the job. The processes (Systems 1) are operationally managed by System 3 by monitoring performance and assuring (System 2) the flow of product between System 1s and out to users.

System 3 is able to audit (via 3*) past performance so "bad times" for production can be compared to "good times". If things go wrong and levels of risk increase the System 3 asks for help or puts it to colleagues for a remedy.

Whilst specialist groups may be used to help provide focus and impetus for the functions of System 4, they should be augmented by a range of integrative mechanisms to align the separate narratives competing for attention.

Includes additional feedback on the true state of operations across the business entities. via independent audits. they must be sporadic and infrequent, since routine audits relinquish much of the variety they generate.? Second, they must be openly declared mechanisms to avoid concerns about excessive corporate control.? Third, they must only link two adjacent levels of recursion to avoid a breakdown in trust or an overload of unnecessary information.

• How are activities to be defined, managed and monitored?

• How will assurance activities be used to verify performance?

System 4 - Outside and future

The Intelligence function is the two-way link between the primary activity (ie. Viable System) and its external environment. Intelligence is fundamental to adaptivity; firstly, it provides the primary activity with continuous feedback on marketplace conditions, technology changes and all external factors that are likely to be relevant to it in the future; secondly, it projects the identity and message of the organization into its environment. These loops must operate in balance, to avoid either overloading the system with a swamp of external research data without the capacity to interpret and act on that data; or the alternative risk of communicating outwards in a strong fashion, without having a corresponding means to listen for feedback from the marketplace.? The intelligence function is strongly future-focused. It is concerned with planning the way ahead in the light of external environmental changes and internal organizational capabilities so that the organization can invent its own future (as opposed to being controlled by the environment). To ensure that its plans are well-grounded in an accurate appreciation of the current organizational context, the intelligence function also needs to have at its disposal an up-to-date model of the organization.

The Intelligence function is the two-way link between the primary activity (ie. Viable System) and its external environment. Intelligence is fundamental to adaptivity; firstly, it provides the primary activity with continuous feedback on marketplace conditions, technology changes and all external factors that are likely to be relevant to it in the future; secondly, it projects the identity and message of the organization into its environment. These loops must operate in balance, to avoid either overloading the system with a swamp of external research data without the capacity to interpret and act on that data; or the alternative risk of communicating outwards in a strong fashion, without having a corresponding means to listen for feedback from the marketplace.? The intelligence function is strongly future-focused. It is concerned with planning the way ahead in the light of external environmental changes and internal organizational capabilities so that the organization can invent its own future (as opposed to being controlled by the environment). To ensure that its plans are well-grounded in an accurate appreciation of the current organizational context, the intelligence function also needs to have at its disposal an up-to-date model of the organization.

Pattern recognition in chess is possible without considering and grading every possible move. A filter should be used that can exploit pattern recognition capable of recognizing patterns in the developing, imminent future. It must acquire relevant criteria from knowledge about the rules (or corporate ethos). The typical strategy one adopts until a depth picture emerges is to strengthen oneself. Remember the decision not to act is a current action.

System 5 - Identity and policy

The last function, giving closure to the system as a whole, is the policy-making function. This function is by definition low-variety (in comparison with the complexity of the rest of the organizational unit and the even larger complexity of the surrounding environment); it, therefore, needs to be highly selective in the information it receives. This selectivity is largely achieved through the activities and interactions of the Intelligence and Control functions.? The main roles of Policy are to provide clarity about the overall direction, values, and purpose of the organizational unit, and to design, at the highest level, the conditions for organizational effectiveness.

Miller argues that living systems exist at eight nested hierarchical levels, not unlike matryoshka dolls. These levels begin at the cellular level and build into organs, organisms, groups, organizations, communities, societies, and supranational systems. Miller argues that each of these levels can be studied from a common viewpoint, that of nineteen comparable subsystems which each interact within their enclosing and subordinate structure to form a holistic system, as Figure 1 depicts. Each of these subsystems operates within a context which is appropriate to the level of the system. Collectively, all achieve transformations of space and time, matter, energy or information, performing functions such as locomotion, manipulation, communications, and expulsion.

Regardless of scale, state, or perspective, these living systems have both form and function and have properties that can be observed and categorized. As these patterns are recognized and decomposed, a foundation for reasoning can be formed on topics such as the best possible arrangement and provisioning of parts for achieving some purpose. To pursue this purpose, each entity must make effective choices from real options available to influence its future, given a set of initial conditions, so that beneficial emergent properties can unfold across space, scale, and time.

He drills into each of these subsystems as both a categorization scheme (or logical architecture) and with a specialist's love of details, revealing both the structure and processes of each subsystem. He also goes to a great deal of effort to explain key characteristics of these patterns across all levels; for example, he devotes one whole chapter (of eighty pages) to the topic of information overload, a topic essential to understanding the forces and constraints at play. He also provides a full-color plate in which he points out common characteristics of these subsystems within the level being studied (three are depicted below).

An Organ

An Organism

An Organization

Highlights

3. Subsystems

3.1 Subsystems which process both matter-energy and information
3.1.1 Reproducer
3.1.2 Boundary

3.2 Subsystems which process matter-energy
3.2.1 Ingestor
3.2.2 Distributor
3.2.3 Converter
3.2.4 Producer
3.2.5 Matter-energy storage
3.2.6 Extruder
3.2.7 Motor
3.2.8 Supporter

3.3 Subsystems which process information
3.3.1 Input transducer
3.3.2 Internal transducer
3,3,3 Channel and net
3.3.4 Decoder
3.3.5 Associator
3.3.6 Memory
3.3.7 Decider
3.3.8 Encoder
3.3.9 Output transducer

4. Relationships among subsystems or components

4.1 Structural relationships
4.1.1 Containment
4.1.2 Number
4.1.3 Order
4.1.4 Position
4.1.5 Direction
4.1.6 Size
4.1.7 Pattern
4.1.8 Density

4.2 Process relationships
4.2.1 Temporal relationships
4.2.1.1 Containment in time
4.2.1.2 Number in time
4.2.1.3 Order in time
4.2.1.4 Position in time
4.2.1.5 Direction in time
4.2.1.6 Duration
4.2.1.7 Pattern in time

4.2.2 Spatial-temporal relationships
4.2.2.1 Action
4.2.2.2 Communication
4.2.2.3 Direction of action
4.2.2.4 Pattern of action
4.2.2.5 Entering or leaving containment

4.3 Relationships among subsystems or components which involve meaning

5. System processes

5.1 Process relationships between inputs and outputs

Matter-energy inputs related to matter-energy outputs

1. Matter-energy inputs related to information outputs

2. Information inputs related to matter-energy outputs

3. Information inputs related to information outputs

5.2 Adjustment processes among subsystems or components, used to maintain stability
5.2.1 Matter-energy input processes
5.2.2 Information input processes
5.2.3 Matter-energy internal processes
5.2.4 Information internal processes
5.2.5 Matter-energy output processes
5.2.6 Information output processes
5.2.7 Feedback

5.3 Evolution - Emergence, growth, cohesiveness, and integration

5.5 Pathology

1. Lack of matter-energy inputs

2. Excesses of matter-energy inputs

3. Inputs of inappropriate forms of matter:energy

4. Lack of information inputs

5. Excesses of information inputs

6. Inputs of maladaptive genetic information in the template

7. Abnormalities in internal matter processes

8. Abnormalities in internal information processes

5.6 Decay and termination

As system engineers are prone to say, the rest is just details, though that would be missing Miller's point. His conclusions emphasize (especially at the level of supranational systems), that it is far better to recognize such patterns and deal with them proactively than to wait until decay and termination set in. This insight relates to our responses to threats such as cybersecurity and related changes in our current social, technological, and political environment, though only if the mitigations are appropriate, timely, and effective.

Economists’ thinking about production has evolved from simple input–output rules to today’s nuanced, multi-factor frameworks. These conceptual shifts include transitioning:

1. From an engineering metaphor to an economic optimization tool

2. From fixed technologies to technology itself as an endogenous, evolving factor

3. From two-factor simplicity to multi-factor, multi-output complexity

4. From static efficiency to dynamic productivity and innovation analysis

This evolution in thinking originally rested on the concept of production as a black box and examination of what powers it. Let’s open this box and see what’s inside.

Means of production

The concept of the means of production, depicted here in Figure 1, is widely used to signify the relationship between things used as inputs to a production system and the constituent mechanisms needed to provide those inputs - factories, tooling, systems, services, and methods - whether in a system of interest, or within an economy or society.

Karl Marx focused on these means of production to distinguish them from contributions by labor. Elon Musk sees them as levers for exponential innovation and automation. Two components of these means - resources and capabilities - deserve further detailing. For orientation for my intended use of the term power, see here.

Resources as power

Production for a system of interest consumes resources provided by upstream providers (who produce them) and produces resources for downstream consumers (who consume them). Energy is one of the most critical resources - the capacity to do work - since it fuels all actions, from powering a data center, lifting an object, or running a machine.

Energy comes in various forms - kinetic, potential, thermal, electrical, etc. Power is the rate at which these forces can be applied in accomplishing work or transferring resources. Think of it as the speed of doing work. It can tell us how effectively resources are being used to accomplish useful work. Energy is a stock, while power is a flow. If unfamiliar with this terminology, see this post.

The three primary resources, as shown in Figure 2, are time, talent, and treasure (capital), as those allow effort to be applied, capabilities to be matured, and objects to be transformed. While the selection of the other resources - forces, information, and raw materials - are important for efficiency, the primary forces enable better objects to be used in assembly, greater competency to be incorporated into capabilities, and better domain knowledge to be leveraged in performing the work.

Capabilities as power

Capabilities (Figure 3) provide the ability to execute specified courses of action, and guide and constrain agents in performing these activities.

The concepts underpinning capabilities have had the greatest reach in military applications today, reflecting how armed forces think about applying power in conflicts while keeping a mission focus. The applications of these concepts focus on preparation - to understand how well forces can reliably employ resources within anticipated situations. This focus goes beyond traditional planning, which focuses on raw numbers of resources and equipment, to consider the capacity to successfully achieve desired courses of action. When considered in this context, capabilities are sources of power, but unlike resources, are not consumed in usage (though they can be degraded with time).

Maladies in production

Many things inevitably go wrong in accomplishing this work, especially when affected by complexity, external influences, resource shortages, quality breakdowns, and garbled communications. In this post we are focusing on the implicit structure for each production operation when encountering these situations. These are only apparent when observing patterns of operations across the requisite variety of contexts. When they arise, they lead to internal pivots in straightforward processing, or off-ramps from further processing. Such maladies are chronic, problematic conditions that introduce disruptions in production equilibrium, whether biological, psychological, or systemic. They often signal:

• A misalignment between internal states and external demands

• Flaws in communications and exchanges

• Breakdowns in incorporating information from feedback loops

• An inability to effectively adjust resource allocations as adverse conditions arise

Let’s examine the many types of pitfalls that are at the root of these production maladies:

1. Shortfalls or incorrect resources (time, talent, treasure, information) provided for the work

2. Breakdowns in preparation (orientation, architecture, stakeholder engagement) or orchestration of required actions

3. Inadequate attention to risks or change management

4. Slips and lapses in performing the work (resulting in defects that require rework)

   1. Mistakes in analyzing what is needed for the work to be performed effectively

   2. Performing work out of sequence

   3. Mistakes in performing the work itself

   4. Failure to properly document the work

5. Externalities adversely injecting change into endeavor

6. Improper use of tools - including automation - to situation

7. Incomplete testing of functions, performance, structures, or failure modes

8. Procedural disconnects (mismatches between situations and domain knowledge)

Each of these maladies threatens the orderly flow of work and thus adversely impacts throughput across each step of production.

The Minimum Viable Production Flow

If we are going to orchestrate production adequately, we must account for all these steps and transformations that occur, track the frequency, pace, and rework of each step, and use that information to learn to apply and mature capabilities and means of production to minimize non-value-added detours through this flow. The underlying steps and feedback loops are depicted in a stock and flow model, in which flows are represented by valve symbols representing transfer functions and stocks are represented as boxes depicting reservoirs. The processing steps are described generically here but are mappable to most situations. Note that inputs and outputs required by each step are not shown here but are implied in separate writings about portfolio management.

Once jobs are committed, the work is distributed across performing agents, depicted in Figure 6 as multiple parallel efforts for each step (when steps indicate depth):

We want our model of these dynamics to be viable for representing as many situations as possible, while being as simple as possible. Performance of production systems is non-linear and classically follows what is described as an S-curve - the sigmoid function whose graph is shown in Figure 7. This pattern is consistent with the elements described in figure 3 of this post.

In Brian Potter’s book The Origins of Efficiency, these dynamics are described as follows:

When we talk about production methods, we’re really talking about technologies… a production process can be thought of as a large collection of different technologies strung together to accomplish a particular goal… For individual technologies, progress tends to follow an S-shaped curve, with time or effort on the horizontal axis and technological performance on the vertical axis.

Early on, the technology performs extremely poorly, if it works at all. The phenomenon at play in the technology may have only recently been discovered and as a result is poorly understood. It might not yet be clear how it behaves under different conditions, or what arrangement of components can best take advantage of it, or even for what purposes it might be harnessed. Fixing one problem with a nascent technology tends to simply reveal more problems, so significant time and effort might be invested without any noticeable increase in performance.

But over time, as scientists, engineers, and tinkerers explore different ways of implementing it, the technology’s characteristics become better understood. As people begin to figure out what works and what doesn’t, the search space of the technology is narrowed, and it attracts more talent and funding. As attention converges on the most promising avenues for advancement, performance improves more quickly… This gradual refinement that leaves the basic nature of the technology unchanged is often called incremental or evolutionary improvement. During this period, the technology might converge on a dominant design: a specific way of implementing the technology that can be easily adapted to serve the needs of many potential users.

Eventually, a technology’s performance approaches some natural limit: the maximum level of performance that the given effect or principle at work or the structure of the dominant design can achieve. As the technology approaches this limit, gains in performance are harder and harder to achieve, and the rate of improvement slows.

As the legend in Figure 6 indicates, the steps in our production process manifest performance through four phases (estimates below are approximations for a typical small to medium-sized endeavor):

1. Inception: the scope, goals, expected benefits, risks, and feasibility of the endeavor is assessed. The effort (~5%) and schedule (~10%) at this point are small but establish the baseline for future acceleration.

2. Elaboration: The requirements, strategy, and structure of the work are explored, and the necessary details are captured, organized into jobs ready for assignment, and baselined. The curve begins to steepen as more resources are engaged, and the definition of jobs and methods stabilizes. This phase involves ~20% of the overall effort and ~30% of the schedule.

3. Construction: As jobs are committed and prepared, actionable elements of the work elements are initiated, work in process is generated, results of that work are stabilized, and solution elements are aligned, with a focus on functionality and performance. This represents the bulk of cost (~65%) and schedule (~50%) consumption, powered by peak learning and velocity.

4. Transition: The solution is validated and made available for delivery, with training, validation, and final adjustments. Effort (~10%) flattens and schedule (~10%) reflects wrap-up activities.

The pathway through this landscape is challenging. Each step’s performance has a probability distribution reflecting the natural variation of processing. This variation is amplified as this variation, and its corresponding limits of capacity, inject uncertainty, incorporate feedback from downstream steps, as these functions interact, and adjust to maladies encountered along the way, rinsing, and repeating with each time increment.

Transfer functions

Transfer functions characterize each action’s output for each input. There will be many complex inputs in addition to what is shown in Figure 6; those collectively influence decisions that are abstracted into the Power value that feeds the rates of each step in the flow. Such functions can be implemented by human agents or agentic transformers.

The steps depicted in Figure 6 are outlined below, with flows (the transfer functions) stylized in bold, stocks in italics, and the transition indicated by an arrow character in the following lists. The normal flow (aka “happy path”) is described in the outermost, numbered outline level. These steps typically have one or more feedback loops that are captured underneath each step as indented, exception processing pathways which inject rework into its upstream steps, resulting in delays.

Step sequences

1. Observation: Signals → Opportunities

   Observation is the act of noticing, watching, or measuring something to gather information. These observations can be passive or active; each involves:

   • Perception: Detecting the signals through sensing and communications

   • Attention: Focusing on specific signals while filtering out noise

   • Interpretation: Deriving meaning from perception free of influence from prior knowledge or bias

   Signals come in many flavors - orders, resource shifts, support obligations, and expedite requests among them - but each requires ingredients in amounts appropriate to each situation. Noise can obfuscate these signals; indeed, when the level of noise exceeds the signal, chaos dominates the search for worthwhile opportunities, and can increase the likelihood of subsequent pivots, or influence decisions about needing to ignore opportunities. Incentives must be sufficient to amplify effort sufficiently to effectively capture the opportunities despite the noise.

2. Orientation: Opportunities → Goals

   Goals are a bridge between intentions and execution. Orientation must validate the feasibility of each opportunity (considering timing, constraints, and achievable throughput), confirm relevance with priorities and values, and translate visions into outcomes that realize the intended successes. Goals should follow best practices and describe what success looks like in unambiguous terms, anchoring opportunities with timing to reinforce urgency, provide room for course corrections, and enable evolution as feedback reveals new insights or conditions.

   1. Ignoring Opportunities

      Not all opportunities are deemed worthwhile to pursue, so some number will out of necessity be ignored. Delays in processing are an indicator that such disposition should be considered.

3. Analysis: Goals → Requirements

   Goals are analyzed and translated into requirements through the capture, refinement, and elaboration of context, intentions, and constraints into specific, testable conditions that must be satisfied.

   1. Pivoting towards different Opportunities

      Some goals are worthwhile in pursuing an opportunity but require revision prior to analysis.

   2. Discarding Goals

      Other goals are not critical to pursuing the opportunity and can be discarded.

4. Organization: Requirements → Jobs

   In his book Competing against Luck, Clayton Christensen describes jobs from a customer’s perspective; progress involves understanding what that customer is trying to do in particular circumstances, and what decisions they must make. This step elaborates statements of work, allocates responsibilities, and distributes requirements across jobs for responsible agents.

   1. Revisiting Requirements with respect to Goals

      As requirements are fully considered, the need for clarification, conflict resolution, and addressing missing elements arises, forcing further work on the related requirements.

5. Commitment: Jobs → Preparation

   Commitments require a psychological and relational progression through

   three phases:

   • Understanding: Gaining clarity about the context and expectations for the jobs

   • Acceptance: Acknowledging and embracing the reality of these inputs and their relationship to the jobs being assigned (including responsibilities, relationships, limitations, and challenges)

   • Support: Agreeing to accept these responsibilities and collaborate upstream and downstream to ensure adequate progress is achieved

   Preparation ensures actions connect to broader goals or narratives and empowers responsible agents to navigate this progression so that work can proceed in a straightforward fashion. This preparation is the traditional role that planning performs and enables parallel action in subsequent steps if sufficient agents and resources are available.

   1. Refactoring Jobs given new Requirements

      Requirements changes can necessitate adjustments to job structure considering those changes.

6. Provisioning: Preparation → Actionable work

   Once the parties responsible have committed to perform the necessary work, preparation is performed to define the steps necessary, required interdependencies, and secure the resources necessary for implementing these steps. Work becomes actionable by provisioning the inputs necessary for execution.

   1. Enhancing Preparation given new Jobs

7. Initiation: Actionable work → Work in process

   Although work may be actionable, many factors can delay launching the effort:

   • Cognitive friction: Complexity can overwhelm; goals and requirements can lack clear definition; and uncertainty around prioritization and sequencing can paralyze action.

   • Emotional and relational factors: Anxiety, low motivation or energy, and insufficient alignment on values can burden progress with negotiation or avoidance.

   • Structural barriers: Poor integration of tools, unclear feedback mechanisms, multitasking, and resource shortages (promised but unfulfilled) can erode progress.

   To optimize flow, these constraints should be addressed rather than passed downstream. Initiating action converts potential into momentum, shifting from planning to doing.

   1. Augmenting Requirements after consideration of Actionable work

      Revisions to requirements are typically triggered by new insights into stakeholder needs, technical feasibility, risk exposure, or strategic alignment. These insights often emerge from real-world feedback, evolving conditions, or deeper understanding gained in implementing these requirements.

8. Generation: Work in Process → Results

   Each action is primed by clear intentions and requirements which set the direction and define what success looks like. Interactions with other actions and related systems feed checkout, incorporate feedback, and enable progress to be accumulated.

   1. Discovering new Requirements while performing Work in Process

      See 7.a

9. Inspection: Work in Process → Reviews

   Reviews are a mechanism to generate signals that inform whether to continue, refine, or pivot work in process and candidate solutions. The entry conditions for such reviews are an important consideration in organizing the work.

   1. Abandoning Work in Process

      Effort invested in preparing for and performing work accumulates; when work is abandoned, those efforts are wasted and contribute to the opportunity costs of the endeavor.

10. Incorporation: Review results into Work in Process

    Typical dispositions of review findings include acceptance, conditional acceptance, revision requests, rejection, and escalation for further action. These outcomes vary depending on the context - legal, academic, regulatory, or organizational - but they reflect the reviewer’s judgment on the adequacy, accuracy, and implications of the material reviewed. Additional actions required to achieve this disposition are added to the work in process.

11. Stabilization: Results → Solution elements

    Elaboration, trial, and error powers the maturation of results into candidate solution elements.

    1. Correcting Results

       When results require correction relative to

12. Alignment: Solution elements → Candidate solutions

    Holistic solutions require that the pieces interact properly and collectively satisfy requirements.

    1. Diagnosing problems with Solution elements

13. Validation: Candidate solutions → Progress

    Each proposed solution must be evaluated against real-world conditions before being released into service.

    1. Isolating problems with Candidate Solutions through Reviews

The gears driving production rates

As Figure 6’s legend indicates, each step can be characterized into one of three categories, using the metaphor of a funnel:

1. Exploration: High Variability, Low Throughput

   In exploration, we are in the wide part of the funnel - the scope of searches is wide, and the entropy is high.

   1. Characteristics

      1. Purpose: Discover options, surface unknowns, generate hypotheses

      2. Inputs: Often ambiguous or incomplete

      3. Outputs: Possibilities, insights, candidate pathways

   2. Throughput Implications

      1. Rate: Low and uneven; progress is nonlinear and often recursive

      2. Variation: High given the low quality of information available; cognitive load, novelty, and branching paths reinforce this unpredictability

      3. Constraints: Bottlenecks often stem from lack of clarity, insufficient framing, or premature convergence.

2. Evaluation: Moderate variability and throughput

   Evaluations narrow the funnel through the incorporation of structured filtering.

   1. Characteristics

      1. Purpose: Compare, prioritize, validate, and assess trade-offs

      2. Inputs: Structured options or hypotheses

      3. Outputs: Ranked choices, decisions, or filtered paths

   2. Throughput Implications

      1. Rate: Moderate. Can accelerate with clear criteria and decision matrices.

      2. Variation: Medium; depends on complexity of criteria and stakeholder alignment

      3. Constraints: Bottlenecks arise from unclear metrics, cognitive bias, or decision fatigue

3. Transformation: Low Variability, High Throughput

   These steps exploit upstream processing by operating as the nozzle of a funnel does, allowing agents to ‘turn the crank’ and deliver at high rates.

   1. Characteristics

      1. Purpose: Execute, build, incorporate, implement, or convert

      2. Inputs: Validated decisions or designs

      3. Outputs: Tangible results suitable for follow-on steps

   2. Throughput Implications

      1. Rate: High. Execution benefits from repeatability and automation

      2. Variation: Low; standardized inputs yield predictable outputs

      3. Constraints: Bottlenecks shift to resource availability, tooling, or integration points

Each exception processing step in the step sequences above are explorations for context by searching for the relevant details of the presenting conditions so the underlying problem can be accurately diagnosed. Transformations and evaluations have their own puzzles to be solved, but the context in both those cases is known - to produce a useful result within the attack surface of the territory and goal in question.

Wrap-up

Now that you understand this DNA of production generically, it should become apparent that even straightforward pathways can quickly become quite complicated. Such complications can be difficult to unwind, especially for the inexperienced. Practice and learning are the best medicine for such dilemmas. Such learning requires understanding each of the writings in this series, animating the above dynamics under alternative initial conditions and injections by externalities, and rehearsing the actions expected by each of the

Figure 1 is a depiction of a real machine which was used around 60 A.D. for removing standing water from underground mines. These machines are known as reverse overshot water-wheels, and one of them was recovered from a copper mine at Rio Tinto in Spain and is now at the British Museum.

We know about these devices from the writings of Vitruvius, the Roman architect. He described a specific installation consisting of thirty-two water wheels, mounted and stacked above one another within the vertical shaft of a Roman mine. Each wheel was turned by one or more individuals who walked along the top of the water wheel to spin them.

A batch of water was raised up a specified distance as each worker turned their assigned wheel. This water then became a queue of work for the next higher wheel and worker to process. Each worker could only consume the output of the prior worker after enough water had accumulated at their assigned stage.

The pace of the slowest worker in the chain would thus constrain the end-to-end throughput potential across the entire device. Despite this limitation, their efforts were collectively able to lift the underground water about 200 vertical feet from the mines where they were installed, which was an impressive accomplishment for that time, and which gave the endeavors who used this technology advantage over other competing approaches, such as a bucket brigade.

This device can be considered from two distinctive perspectives. The first viewpoint consists of its capability to raise a column of water by a particular distance; we'll call this a water elevation system. The second perspective uses such a capability in the context of a larger ore extraction system; its primary goal is to economically produce ore which can then be provided as a resource for many other purposes that are not relevant to our example. The extraction of ore was financed by investors who saw this potential. Not all mines had water draining into the tunnels being mined, but when it did, the water represented a constraint which blocked the potential realization of value that could be achieved.

The economics of such ore extraction is determined by the value that ore has in the marketplace, the number of resources necessary to produce that ore, and how long it takes to produce it. Each mine would have unique characteristics, such as the volume of water flowing into the mine, and the height to which the water needed to be raised. The operation of an ore extraction system is only economically viable within a performance envelope and operating context which each device was designed to serve. The throughput of the water extraction system had to be sufficient to consume inflows from all sources of water and any accumulated pool of water present at startup.

The ore extraction system and the water elevation system are interdependent since each requires the other to provide value. The costs to develop and operate the water elevation system will be bounded by the economics of the ore and its transport to a location where the ore is used as an input to another system that would consume the ore, such as a smelter.

The water elevation system would require water to be drained sufficiently so that ore could be extracted from the mine. One can envision water seeping into these underground mines from many sources, such as from rainfall, springs, or underground streams. Each of these inflows may have a varying rate over time, such as when a sudden rainstorm occurred. It might not take many people to extract water during a dry period, and it might take a long time after a rainy period to remove sufficient water for the ore extraction to be resumed. Delays in developing or operating the water elevation system could lead to delays in the time it takes to perform ore extraction, and thus significantly reduce the time-adjusted value of that ore. The actions and skills required to produce operational outputs are fundamentally different from those necessary to develop new capabilities that could extract water more efficiently and effectively.

In this example, the ore is an output of the mining value chain. The water wheels which were used to raise water required personnel, equipment, information, and tools to realize the needed capability. The capability of a water wheel to remove this water would be determined by many factors, including the volume of water which each wheel could consume, the pace at which workers could operate the system, and the leakage which occurs across the elements of the system itself. During operation, pools of water formed at the boundaries between each processing stage performed by each wheel. The size of these pools can provide a visual indicator of the backlog of the work cells that are accomplishing that work. Such queues are inevitable in any development or production process, due to variation in the quality of requirements, the capacity of team members, and the information available to the team.

Development environments share most of the features of the water elevation system described above. In development settings, a cascading process is typically used, as depicted in Stuart Pugh's core design activity model of the interactions of technologies and technique shown in figure 3; note that in this case, the vertical flow is depicted top to bottom, unlike in the water wheels, which are subject to gravity. Each stage requires horizontal integration of the elements relevant to the structure of activities and material relevant to that stage, though the translation into wedges of activity depicted may vary by stage. The flow among team members involved in such designs can be thought of as like those of our cascading water elevation system.

The overall throughput of a team is a function of the interactions between the team members. Feedback can be used to manage the rates of injecting work (be it new or rework) into the system. The queues of work in process between team members can be used to provide visual indicators of the flow of this work as it moves through the system. The size of these queues will be dependent upon whether a batch production, job production, or continuous production technique is being used. A description of managing such flow is here.

Production decisions regarding the allocation of resources across work stages are often driven by the robustness of the forecasting information available, the cycle time variation inherent in each operation, and the quantity necessary for efficient economic operations.

The size of each batch to be processed across each stage of such an operation is critical for many reasons:

• Communications costs grow as connections are n**n for n units, and many interactions are often necessary to coordinate the work across these channels

• Large batch sizes limit efficiencies that may be achieved by performing similar and routine operations together

• Large batch sizes also reduce the usefulness and effectiveness of feedback to improve the system, which complicates connecting cause and effect when something goes wrong.

The value of a batch of work can only be evaluated within the context of the environment within which that work is being performed. A tool (like the water wheels used to provide the capability to elevate the water in question) is only a means to an end; in the case of mining, the immediate mission is to extract marketable ore from geological deposits. Far too often, we are inclined to measure value by quantifying the resources which are visible, or which are easy to measure, rather than by the flow of this work in the context in which value is realized. This can cause us to measure work by the number of people who descend into the mines each day, rather than considering the efficiencies being realized in the various stages of the flow. As Donald Reinertsen indicates:

Product developers create a large number of proxy objectives: increase innovation, improve quality, conform to plan, shorten development cycles, eliminate waste, etc. Unfortunately, they rarely understand how these proxy objectives quantitatively influence profits. Without this quantitative mapping, they cannot evaluate decisions that affect multiple interacting objectives...

Life would be quite simple if only one thing changed at a time. We could ask easy questions like, "Who wants to eliminate waste?" and everyone would answer, "Me!" However, real-world decisions almost never affect only a single proxy variable. In reality, we must assess how much time and effort we are willing to trade for eliminating a specific amount of waste.

A modern, kid-powered version of one stage of this water wheel is in use today for pumping water in underdeveloped countries. In field use, such devices have often failed to meet expectations, and have proved to be more costly, less reliable, and harder to use than the child's play they were marketed to be.

Regardless of the outcomes expected from such production systems - be they technological maturity, new product development, or service excellence - value can only be realized once solutions have been completed and have been placed into service. This can only occur at the rate of acceptance of the methods by users and the throughput of the system overall.

Until then, investments are just holding costs for an uncertain future. They merely represent an inventory of work which has begun but which has not yet been completed. This inventory can be particularly large for software-intensive efforts since the inventory is less tangible than it is for physical assets. This means that investments in labor may exceed the costs of materials, especially when the costs of paying for this labor pool over time are made visible. The size of this inventory is equally critical, since the value they offer may become stale over time, as opportunities for usage and customer enthusiasm for requested work tend to evaporate during dry seasons.

A learning organization can incorporate these performance insights into behavioral changes quickly, efficiently, and effectively. For this learning to gain traction, an organization must design affordable interventions, mitigate constraints arising in different situations, and channel information to where it is needed. These practices are more likely to get traction when they emerge from the collective insights of team members rather than appearing as mandated interventions from above.

The first operational goal of any endeavor should be to lay out the steps required for the realization of end-to-end flow, emphasizing the cyclical nature of these operations. The total elapsed time between the beginning of the first processing step and the end of the last processing step of these iterations is the cycle time. Within development processes, productivity improvements are always a function of achieving cycle time improvements in some form.

Waterfall-based development processes provide recipes for large-batch processing. They share the same wastefulness, inefficiency, and risk as bureaucracies. Therefore, they tend to accumulate large inventories of work-in-process at each step in production and suffer from decreasing throughput over time. A strategy of small-batch, continuous flow can instead identify opportunities to improve efficiency by eliminating the wasteful burdens of delay, work in process inventory, and communications breakdowns between process steps.

We can highlight these principles by performing a thought experiment within the context of transportation planning. The purpose of highway systems is to enable the efficient transport of people and goods from one place to another. The total resource consumption and cycle time of such systems are a function of traffic congestion since pathways are susceptible to the tragedy of the commons. Traffic flow over these pathways is a function of the speed and density of the vehicles and is. As traffic density approaches the capacity of any segment, average traffic speed at that location will begin to decrease.

It is easy for bottlenecks to arise in such systems. When traffic density begins to exceed a particular loading factor, local perturbations (such as a driver touching their brakes) can cause a cascading effect over a large distance, and this delay can propagate for a long period of time. While one delay is playing out over time, the probability of a second issue arising is increased and may have an even greater impact.

Predicting the time it takes for a particular vehicle from one location to another is highly dependent upon many factors. Average flow is impacted by many factors - the average capacity of the channels over which it travels, the patterns of demand at different points along those pathways, environmental factors, bottlenecks, characteristics and interactions of the vehicles, etc.

The nature and quality of information available to manage such flow changes with time, Further, delays often impact other parts of the endeavor, resulting in further waste and inefficiency. Yet this flow cannot be increased just by encouraging each vehicle's driver to maintain their speed. Because of the frequency and impacts of these constraints, infrastructure improvements should be deliberately designed to enhance flow. For example:

• Incoming lanes are designed to allow entering vehicles to merge at the same speed as the rest of the traffic.

• Metered ramps throttle vehicles on entrance ramps carefully to maintain spacing between traffic.

• Emergency vehicles are pre-positioned to respond quickly to problems.

Individual drivers desire minimal transit times, yet lack traction to do something about it themselves; the system they are in just pushes back, in ways that can frustrate everyone.

When resources like gasoline consumption are considered for this transportation system overall, the impacts of relatively minor issues can thus have major systemic effects and result in far greater impacts than would be immediately apparent. Yet the root cause that caused such effects are often not easily seen from ground level, as the volume of traffic, and the passage of time introduces significant noise into making these associations. Since new designs for roadways take a long time to change, by the time they are actually implemented, they are often no longer relevant to the new traffic patterns which have emerged since these designs were introduced.

This traffic flow analogy helps to explain situations in which processing time is largely predictable, and there is little variability from item to item. This is the case in most manufacturing environments, and organizations usually strive to drive out all waste that arises during production operations within such settings. To return to our traffic analogy, in such environments, each vehicle would only put as much gasoline in their car as they think will be necessary to get to their destination on average, with a small contingency. When there is little variation item to item, this might be an acceptable strategy. Unfortunately, these conditions rarely exist in other kinds of projects and would lead to disabled vehicles clogging up the roadway (and additional vehicles servicing those vehicles). In environments where work differs widely in form, context, and structure, the time it takes to process this work will be highly variable.

Legal recognition by the certification authority that a product, service, organization or person complies with the requirements. Such certification comprises the activity of technically checking the product, service, organization or person and the formal recognition of compliance with the applicable requirements by the issue of a certificate, license, approval or other documents as required by national laws and procedures. In particular, certification of a product involves: (a) the process of assessing the design of a product to ensure that it complies with a set of standards applicable to that type of product so as to demonstrate an acceptable level of safety; (b) the process of assessing an individual product to ensure that it conforms with the certified type design; (c) the issuance of a certificate required by national laws to declare that compliance or conformity has been found with standards in accordance with items (a) or (b) above.

There are many unique challenges associated with the development of such safety-critical software. The projects which must produce this kind of product typically have to deal with issues such as constrained resources, novel system architectures, and real-time performance constraints; these amplify the already difficult challenge of successfully developing software within fixed time allocations and to very demanding requirements.

The introduction of regulatory oversight and conditions into this environment can either help or hinder achieving these goals, depending upon how effectively and carefully this oversight is implemented. If insufficient attention is given to key details during project execution, or if there is a lack of understanding of the underlying requirements necessary for implementation, the risk to the resulting system development activities and products will be high.

Purpose

The purpose of DO-178's guidance is to mitigate these risks by providing sufficient information to successfully plan and implement development activities so that certification can be achieved. When experienced software certification professionals and airframe manufacturing applicants both adopt common guidance, it helps them make consistent determinations of compliance regarding design, development, and operational practices for aircraft and related ground systems.

DO-178 is neither a software development process nor a prescriptive standard for such a process. Instead, the document serves to structure and guide negotiations between applicants and certification authorities on the activities and deliverables which must be accomplished during product development and support activities. The guidance strives to balance the need for tangible material that provides a consistent basis for these determinations against the range of existing airborne software and company processes that are used in actual practice across the industry.

Interfaces

As a result, while the document provides key compliance criteria, it does not provide specific, useful information on how to apply this information on development programs, in a cookbook, 'how-to' manner. The document will thus only be effective when correctly interpreted and applied by experienced professionals, as they prepare, approve or recommend approval of technical data for submission to certification authorities. This style is consistent with the mentoring and selection process for Designated Engineering Representatives across the industry, and fundamental to Organization Designation Authorizations now in place.

The complexity and extent of the compliance data required by DO-178 depend upon the characteristics of the system/software, associated development practices, and the interpretation of DO-178 guidance, especially when it must be applied to new technology and situations in which there is little or no precedent available to draw from. Two different Airworthiness Representatives, working from the document, may still reach different decisions about the acceptability of a particular artifact, because of their experience, preferences, and situation; no standard can or should eliminate technical judgments by qualified agents.

Concepts of operation

Like any standard, each version of DO-178 had good points and bad points (and some versions even contain a few errors). However, careful consideration of its contents, combined with solid engineering judgment, is intended to produce better and safer software. The material is not always popular or easy to implement, but neither is the challenge of assuring the safety of flight. Insights about how the guidance is applied in the real world are also fed back to accomplish continuous improvement of the standard itself.

As an example of how criteria are invoked, some believe that the enhanced rigor and completeness of verification evidence required by DO-178 represents a significant cost burden on the avionics industry. Representatives have complained that producing this evidence requires an inordinate amount of effort, and question the value resulting from this effort. Such views must always be considered in the context of the standard's mission. Since the original emphasis of the DO-178 document has always been safety rather than cost-effectiveness, companies will naturally have widely varying approaches to implementing DO-178, and some are clearly more cost-effective than others.

Call to action

The initial version of the document was published in 1982, just prior to certification of the first models of Boeing's 757 and 767 aircraft. When the first revision, DO-178A, was published only 3 years later, the avionics industry had just completed a major transition from analog to digital systems, in which software played a much larger and more prominent safety-critical role. As a result of this transition, many more companies were subject to certification oversight and compliance determinations. During this initial use, an unacceptable portion of these companies struggled to successfully certify their systems, because these versions of DO-178 were not sufficiently detailed to provide them with sufficient implementation guidance.

Because of these shortcomings, the industry quickly came to a conclusion that still another version of DO-178 was needed. The new version was expected to incorporate experience from these initial applications of prior versions of the guidance and to address a number of questions that were being raised as a result of emerging technical issues and proposed new approaches. In some situations, these questions required new rules to be established, such as in identifying mechanisms for certifying reusable software components, qualifying tools, assessing object-oriented design impacts, and dealing with inoperative code in operational systems.

Emerging drivers

To pick just one of these examples, DO-178 activities are expected to produce rigorous and complete verification and traceability evidence, and the efforts required to produce this information can be quite substantial. As the complexity of software has increased, the cost to produce this verification information has also grown proportionally. While tools have always been used in many situations during development, emerging tools were envisioned to play an increasing role far beyond what had previously had been attempted. This reliance upon such tools, potentially with limited reviews of their results and artifacts, motivated the industry to consider a more explicit definition of tool qualification criteria, and to integrate those criteria into the development practices and descriptions of the rest of the document.

The DO-178B revision was also expected to align its concepts and terminology with emerging guidance for systems development which was concurrently being developed. This guidance, documented in DO-254, provides criteria for design assurance of airborne electronic hardware and describes the expected information required throughout project conception, planning, design, implementation, testing, and validation. For example, through joint agreements between the two standards efforts, it was agreed that the terminology used to describe the criticality of safety considerations needed to change from "critical, essential, and nonessential" to "catastrophic, hazardous/severe-major, major, minor, and no effect". To reflect these changes, the software levels which had originally been described in DO-178A as "Level 1, Level 2, and Level 3" would also need to be changed to "Level A" through "Level E" in DO-178B guidance. This change enabled correspondence between system and software levels to be made when accompanied by appropriate system design and implementation techniques that were used during development.

More significantly, a means of clearly defining and communicating the effect of these changes needed to be established for these corresponding requirements as these levels changed. For example, with respect to the types of verification test coverage required for Levels C, D, and E software, only statement coverage needs to be achieved, but for Level B software, decision coverage is required, and for Level A software, each part of a condition must be demonstrated to correctly affect the outcome through the program’s logic. The standards body needed to determine how these differences would be defined, and the document had to delineate them in a way that was clear, concise, and appropriate to the situation.

After consideration of all of these changes, it became apparent that a complete rewrite of DO-178A was required, despite the fact that this would be the third revision developed for this community in only 5 years.

Because of concerns about the direction which this updated version could take, hundreds of participants from across industry were involved in authoring, reviewing, and integrating the resulting DO-178B content. Organizing their efforts required developing mechanisms so that key issues could be surfaced and analyzed, alternatives weighted, and specific text for proposed guidance could be proposed so that each community of interest could form, storm, and norm there way towards consensus viewpoints.

There was significant pressure to 'get this one right'. In light of the volatility of prior efforts. With the benefit of hindsight, it is worthwhile reviewing the keys of success to this effort, which collectively enabled us to successfully manage the scale and nature of participation for the overall standardization, and to improve the quality and usability of the resulting guidance, and its acceptance by the affected stakeholders:

There was skin in the game for the participants

The development of aircraft systems and the software embedded in these systems rely heavily upon a 'safety culture' that actively focuses on producing software with dramatically fewer safety-related defects than traditional practices would normally produce. In such cultures, with lives and company reputations at stake, the opinions of practicing colleagues tend to be much more highly valued than outsiders. The engineers involved in these development efforts must pay meticulous attention to detail, and act deliberately as safety-related requirements and issues are identified. As they do this, other needs like cost-effectiveness, schedule predictability, or adoption of the latest technology innovations, while important to the business, are understandably less important than the focus of DO-178B, which is on assuring the development of software which will operate in life-critical systems.

In this situation, the committee participants recognized that the guidance which would be produced by committee activities would be of primary importance when performing their duties in their 'day jobs' but would not be the only guidance they would need to deal with. While DO178 is described as guidance throughout the document, this is because regulatory authorities describe it as an acceptable means of determining compliance for aircraft certification, without closing the door to other means. In all other ways, however, this document provides the minimum acceptable provisions (i.e. the standards) which must be taken into account in planning and implementing software-based development activities for products that are to receive regulatory approval.

For many other industry standards, such a 'mandate for use' is either not in place or is not achievable because of the way the guidance has been expressed or invoked. Further, compliance in these other situations often does not need to be demonstrated in any formal sense (though groups may still claim they are compliant, without saying by how much), or to whom and how that demonstration has been secured and recorded.

Decisions about the content of DO-178B in this context were made by striving for consensus by all participants. Consensus does not mean agreement, but rather acceptance, and requires a give and take by all participants to find common ground and jointly strengthen the negotiated position. Achieving this consensus across hundreds of interrelated issues required many meetings to work through but paid off in the quality of the resulting document and the endorsement by the broader community. When consensus could not be achieved, minority decisions were documented and were accepted by the Special Committee chairs, but this path was only found necessary in extremely limited cases, and all these concerns were resolved by final publication.

Community engagement

Such consensus-based standards may be biased towards guidance which retains the ability of industry members to refine (rather than replace) their current practices and require judgment by experienced individuals to make detailed compliance determinations. To reinforce, although a standardized format for documentation might be preferred by regulatory groups, any such format selection would likely favor some companies or groups over others. In a team of competing industry participants, any such specific recommendation would favor some over others and thus would fail to achieve the necessary consensus buy-in. The relevance of the resulting guidance with respect to the diverse implementation approaches used over this period is a direct consequence of the team environment which was fostered as the standard was developed and ratified.

Over time, this requirement for compliance demonstration has resulted in forming a community of practitioners from many different perspectives. These practitioners can come together and have meaningful discussions about development practice trade-offs and their consequences. Such trade-offs are a part of the life of every system development. At the end of the day, DO-178 and its processes don't certify safe systems; this community does. The importance of their engagement in this reflection and self-regulation cannot be over-emphasized, and such involvement by practitioners has often been missing from other efforts to crafting a body of knowledge?for other occupations or industry segments.

Terms of reference were established to scope and provide accountability for the effort

Terms of reference were established for the DO-178B Special Committee by the RTCA board to focus and stabilize the efforts of the working groups. These objectives called the group to:

• Examine existing industry and government standards and consider for possible adoption or reference, where relevant

• Assess the adequacy of existing software levels and the associated nature and degree of analysis, verification, test and assurance activities. The revised process criteria should be structured to support objective compliance determination

• Examine the criteria for tools to be used for certification credit - for example, tools for software development, configuration management, or verification

• Examine the certification criteria for previously developed software, off-the-shelf software, databases, and user-modifiable software for the system to be certified

• Examine the certification criteria for architectural and methodological strategies used to reduce the software level or to provide verification coverage, for example, partitioning and dissimilar software

• Examine configuration control guidelines, quality assurance guidelines, and identification conventions, and their compatibility with existing certification authority requirements for type certification, in-service modifications and equipment removals

• Consider the impact of new technology, such as modular architecture, data loading, packaging, and memory technology

• Examine the need, content, and delivery requirements of all documents, with special emphasis on the Software Accomplishment Summary

• Define and consider the interfaces between the software and system life cycles

• Review the criteria associated with making pre- and post-certification changes to a system

• Consider the impact of evolutionary development and other alternative life cycles to the model implied by DO-178A

This direction served as the charter for the team's activities; these terms of reference were reviewed, revised, and accepted by the entire Special Committee before being finalized. This helped to assure that the wording of this charge was understood and accepted by the members who would be implementing this mandate.

Working group responsibilities for developing content were defined

When dealing with hundreds of participants from many different sectors - regulatory agencies, airframe manufacturers, and suppliers - and many perspectives - systems engineering, quality assurance, software engineering, and management - it was necessary to organize committee activities into parallel groups. A working group structure was established so that participants could self-select where they could best contribute. Working group chairs were then selected who were committed to working together to resolve cross-group boundary issues, and who could provide the necessary leadership and guidance to help drive issues to closure within each of these sub-groups. I was the leader of the working group responsible for change management, coordination of key concepts, and integration of guidance.

Governance to manage information exchanges and decision-making was established

Governance is often an afterthought when groups are themselves developing?new governance. However, because of the number of people involved in developing and reviewing DO-178B, effective governance practices were essential to establishing trust and managing the evolution of the material being developed. A table of contents was established early on to describe the intended architecture of the document and provide a basis for assigning responsibilities to the working groups. A template for working papers was also established as a means to identify, track, and communicate progress on critical technical decisions within each working group over time, in a trade study like environment. Key concepts and features that affected multiple sections of the document were then identified, to ensure that agreed-to approaches for these concepts were developed before tackling other related issues. For example, the group wanted to avoid specifying any particular structure for required artifacts and documents that were to be produced during a DO-178B-compliant development effort, and instead of using these traditional artifact descriptors, made a conscious commitment to describe information in abstract containers that could be implemented in a variety of ways; this decision rippled through many other sections of the material that was developed. Finally, a means of identifying and tracking issues for existing content was established. This mechanism satisfied multiple purposes for the teams' authoring efforts and served as problem reports, change requests, and requests for the resulting information. The communications, tracking, and disposition of these issues were provided centrally in support of all working groups as a service for all participants.

A terminology foundation was documented and maintained

Different groups, and individuals from different backgrounds, often use this different wording in developing content they are familiar with, even though they had the same underlying meaning in mind. These same groups, in somewhat different circumstances, may also use the same wording even though they intended their wording to mean different things. Such communal differences needed to be resolved in order for the resulting guidance which they produced to be coherent.

As a classic example, consider the terms 'verification' and 'validation', which traditionally have many different definitions and interpretations. In DO-178B, verification is defined to be the 'evaluation of the results of a process to ensure correctness and consistency with respect to the inputs and standards provided to that process'. In DO-178B, verification is one of the integral processes (configuration management, quality assurance, and certification liaison), which means it cuts across all the development processes, rather than being a serial activity which begins after a predecessor activity. Validation is defined as 'the process of determining that the requirements are the correct requirements and that they are complete'. Since validation is considered a system-level activity, it is not explored further in DO-178B. Definitions such as these form a solid foundation which is used to tie the other sections of DO-178B together but need to be established upfront so that the content built upon these definitions could be integrated together with a minimum of rework.

Such verification is a part of the checks and balances which DO-178B provides to protect against mistakes. Additional protection is required when this verification must be performed independently, which is defined in DO-178B as follows:

Separation of responsibilities which ensures the accomplishment of objective evaluation. (1) For software verification process activities, independence is achieved when the verification activity is performed by a person(s) other than the developer of the item being verified, and a tool(s) may be used to achieve an equivalence to the human verification activity. (2) For the software quality assurance process, independence also includes the authority to ensure corrective action.

Once definitions such as the above were established, key concepts could then be built from this foundation. As an example, consider the document's treatment of software requirements, which are described as "a description of what is to be produced by the software given the inputs and constraints". They take two forms in DO-178B: 'high-level' and 'low-level' requirements. High-level requirements are defined as "software requirements developed from analysis of system requirements, safety-related requirements, and system architecture", and low-level requirements are defined as "software requirements from which source code can be directly implemented without further information". This definition can obviously be interpreted differently (and thus require further elaboration of low-level requirements) when applied to a new-hire and an experienced engineer, which could drive up costs unnecessarily if this elaboration proves unnecessary once the coding responsibility is assigned. The importance of independence in verification is also highlighted in this situation to ensure that the low-level requirements are taken to the required level.

Development process objectives and interfaces were baselined

Compliance with DO-178B is intended to establish confidence in the activities used to develop safety-critical software and assuring that these activities are producing information suitable for certifying the systems in which the software is embedded within. DO-178B's life-cycle processes are not intended to prescribe or limit the structure of the actual development processes which are put in place by development organizations; instead, the processes described in DO-178B are a descriptive vehicle for communicating expectations about the activities actually performed during development. While these actual activities may take many different forms, DO-178B defines its own set of software life-cycle processes so the actual activities can be traced to the DO-178B processes.

A set of objectives for the DO-178B lifecycle processes are also defined. These objectives define the 'intent' behind these processes and thus describe the value which each of these processes is expected to produce, in a manner like how a judiciary may consider legislative intent in determining their rulings. The objective evidence necessary for demonstrating that these objectives have been accomplished is also described. For example, testing objectives are defined as showing that requirements are fully satisfied, and demonstrating 'with a high degree of confidence that errors which could lead to unacceptable failure conditions... have been removed'. There are obviously many different approaches that can be employed to achieve such objectives, but the proof of their pudding is whether they satisfy these objectives or not. DO-178B objectives also help verify the correct implementation of safety-related requirements that flow from the system safety assessment.

Compliance data is intended to be produced as a natural product of these activities, but unlike many other standards, DO-178B does not prescribe how these activities should be structured. Interpretation and judgment need to be applied in determining the activities and deliverables appropriate to each situation in order to satisfy both DO-178B process objectives and criteria, as well as any business objectives that must also be satisfied (which are outside of the scope of DO-178B).

An evolutionary development plan for the document was outlined

Concepts in any standard need to be developed and communicated in an organized and sequenced fashion so progressive elements can be used to develop the reader's understanding as the document unfolds. A top-level document structure was derived from a consensus view of how certification programs would be described. An outline of the overall document was established, and target page counts were established for different sections, with the overall objective of keeping the length of the document to approximately 75 pages. The thinking was that more writing does not generally lead to clearer writing.

This content for this structure had to be developed incrementally by the responsible workgroups, and this material thus evolved over many meetings in which the working group members held the majority of their team's interactions. As content for the document progressed through position paper development to working-group agreed-to proposals for content and presented that content to plenary, it was apparent that the underlying concepts often did not yet 'stand-alone', and had to be explained by the working group chairs to the rest of the plenary members.

For example, a key change was made in DO-178B in how requirements for software verification were to be defined, with a greater emphasis on requirements-based testing than in DO-178A. This change was intended to discourage the practice of focusing software testing primarily to achieve structural (i.e., White-box testing) coverage, at the expense of fully testing functional and performance requirements. Requirements-based testing - augmented by a determination of the structure coverage achieved by these functional tests - is a more cost-effective and meaningful way to conduct such verification testing.

The emphasis on requirements-based testing resulted in a new and significant focus area in DO-178B, which expects consistency between requirements, code, and the tests that verify that the code satisfies the requirements. Traceability is often used to provide evidence of this consistency and completeness, and such traceability is required to be thorough and bidirectional, demonstrating that requirements trace forward to code and tests and satisfy requirements, and trace backward from tests to code and to the requirements for which the tests were created. At the higher levels of safety and criticality, DO-178B requires that this traceability evidence show 100% structural code coverage. These concepts were each developed within different working groups and were only able to achieve consensus after multiple iterations in the document test - to introduce the idea, integrate its expression, and mature its elaboration. A roadmap of these concept developments and refinements over time (which initially were communicated in the form of working group papers) was helpful to begin developing realistic projections for how long DO-178B would actually take to complete.

A repository was established for sharing information and status

An accessible means of exchanging and providing access to information was put in place to support the committee activities. This mechanism helped improve configuration control of document elements, provided protected access to working groups for the working group papers that were not ready for broader review, and helped build a community of practice to identify concerns and leverage community knowledge in trying to address them. The repository that was established was funded in a creative manner, with software licenses paid by one airframe manufacturer, the hardware paid by another, and the server and associated operations provided by a university that was actively involved in the committee activities. This repository has been so useful that has been upgraded many times and remains in use today.

An editorial team was established to integrate content into a coherent product

While the content of the document was developed by many individuals, it was important for the resulting integrated content to be written as if it had been the work of a single author. Since the document was to be international (and would thus need to be translated into other languages), care needed to be taken to assure that the language and structure of the document were amenable to this translation and that it could be accomplished without changes in the underlying meaning. As an example, the process of progressively refining requirements was described by some authors as 'decomposing requirements', but such terminology, while often used within the United States, would not translate well into French (essentially becoming 'rotting requirements'. Finally, the team needed to establish trust with the original content providers, who wanted the intent of their original writing to be preserved. Over time, what this meant was that responsibilities for the document sections transferred from individual working groups to the editorial team for final integration.

A formalized review process was employed to assure quality criteria were realized

The editorial team employed a structured review process, including a specific checklist designed for such reviews. As these reviews identified issues with the document, the working groups were initially responsible for addressing these issues and determining if the resolution required a change in a technical concept, or how it was explained. As the document entered its final form, change authority was delegated to a smaller number of individuals, to make sure the tone and style were consistent, as the number of ‘must fix’ open issues was driven to zero.

Results

Work on position papers for the document began immediately after the first meeting, using the working group structure that was established in plenary sessions. The first drafts of DO-178B began to be produced in 1989. Work continued through quarterly meetings, with the intervening publication of document updates, exchanges of issues and working group papers, and progressive refinement and convergence on the agreed-to guidance. The final release was adopted in 1992 after the responsibility for the document text was transferred to an editorial board. This team, with representatives from other working groups, conducted intensive sessions to thoroughly review the document, reconcile outstanding issues, and prepare the document for publication. The resulting product has achieved wide acceptance and remains in active use after over thirty years on all new and derivative commercial aircraft developments around the world. The document's usability, clarity, and compatibility with other standards have been well received.

This is one of the proudest accomplishments in my career. I served as the document’s architect, led the editorial group’s efforts to integrate the text produced by all the working groups, and massage their drafts into a holistic document. I received this much-appreciated note (Figure 2) from the overall cochair of the effort acknowledging my contributions:

A new revision, DO-178-C, was initiated in March 2005 and was completed in 2013. Refinements clarified and refined the definitions and boundaries between the key DO-178B concepts of high-level requirements, low-level requirements, and derived requirements, and incorporate the exit/entry criteria between systems requirements and system design (described in ARP-4754A). Since this material has now been widely used in practice for over thirty years, the above practices collectively offer advantages over less formalized approaches that have been used in other standards efforts but are applied in practice very infrequently.

Practices are only effective when used by the right people and the leaders and contributors to DO-178B can remain proud of their contributions to the practice of software engineering in safety-related systems. The above practices are offered in the hope that this evolution can produce a similarly enduring legacy in future work.

For technical debt to be retired, and solutions to be produced more cost effectively, systematic, repeatable methods must be adopted that will produce more reliable outcomes for larger and more complex efforts.

Structured processes were initially introduced into such environments to mitigate the painful lessons associated with more ad-hoc methods. Yet an emphasis on process is frequently cited as a burden by developers that constrains them from pursuing their passion, and getting that which needs to get done, done.

Many have the perception that in-process quality doesn't matter, and that it is the ends, rather than the means, which matters. This is like saying that if water is clean, it doesn't matter where it has come from or what has been done with it. Today, countless development and manufacturing standards have been refurbished to attempt to address contaminants discovered after all processing is complete. Too often, organizations have adopted such standards as protectionist legislation intended to keep new competition out, rather than to guarantee higher quality outcomes.

Some quality definitions

To think through how to do this, some basic definitions are in order. A defect is anything that must be changed in a design or implementation because it detracts from the system's ability to meet stakeholder's needs completely and effectively. Defects can be identified, described, and counted; they are discoverable. The interrelationship between a defect, and the impacts from that defect (which may manifest through failure, as described below), is complex. Even simple defects can cause large problems, and not every defect can result in a failure (since it can be masked by the architecture, or latent until the required combination of events excites that flaw).

When agents make mistakes that result in defects, these lapses are typically described as injecting defects. The rate at which they do this is measurable and has been found to be between 50 and 120 defects per thousand lines of code. Half of these defects can be detected by modern tools during coding and unit checkout; an acceptable proportion of the remainder must be removed through down-stream evaluation activities. If those prove inadequate, they risk escaping into service and affecting operations for customers.

A bug is an observable anomaly which results from a defect and is discovered because an inconsistency with expected behavior is noticed by someone (who hopefully then records this observation so that it can be analyzed and traced to root causes).

Bugs typically manifest themselves through the following pattern:

1. A defect is introduced into a component, or into a related artifact (requirements, design, construction, or test) that is being produced during development. Note that the agents responsible for the component, the artifact, and this verification are often different individuals.

2. When a path through software logic (either within or across modular units) triggers this defect, one of two possibilities arises

   1. the execution state of the code will no longer valid relative to the intentions of the design

   2. the artifact will mislead those responsible for implementation or verification

3. In the case of defects, this invalid state can itself propagate throughout the system in other ways (as other execution sequences are traversed) to produce other invalid states

4. One or more of these invalid states may then eventually result in a bug - the externally visible condition - though this is dependent upon the combination of the execution sequences that are traversed.

5. A failure may further occur when one or more errors in the system occur, are propagated, and cause the system itself to enter an unsafe, incorrect, or unacceptable operational state.

The relationship between defects and failures is thus quite complex and is highly dependent upon the architecture of the system and the robustness of its implementation. Frequently, it isn't just one defect or error that leads to a failure, but a combination of several.

What it takes

Development processes require diligence, effort, and appropriate tools to shortcut routine operations and help us avoid making slips; unfortunately, both people and tools are fallible. Would you drink water from a glass that was dirty, or drink from a catch basin that had a visible scum coating the inside of it? Probably not unless you had no other choice. Yet this insight can clarify the importance of having a well-designed process to produce a high-quality product. Believing that dirty pipes can produce pure water amounts to accepting the position that flawed tools or oblique human efforts will produce satisfactory results without accountability. Still, this myth lives on in the minds of both novices, some experts, and particularly bureaucrats.

Quality is difficult to measure directly. The operations necessary to transform inputs into acceptable outputs require multiple steps to be performed, across diverse environments, regardless of whether a business’s products are suitable for human consumption (like drinking water and software) or not.

If we stick with this plumbing analogy a bit, we can consider requirements to be like a supply of drinking water. The value stream that turns these requirements into useful progress is analogous to the links of pipe that fit together to carry water from its source to where it will be used. Each link is like the steps in the processes of a value stream. These processes may require defining what kind of product is needed, developing artifacts - designs, tests, and verification activities -which assure that the product complies with its requirements and does not introduce hazards to customers. If the links don’t interface properly, or there are missing links, value will be difficult to realize.

If the interconnections between these activities are like catch basins, buckets, and funnels, then someone must make sure the aggregated flow accumulating in these reservoirs is not stagnant or contaminated. Eventually something will exit from the pipeline, and with perseverance, it will be a quality product. The integrity of the pipeline will impact the quality of the output. There is no guarantee that even the most superb development processes will produce correct products and services.

Software quality has become a catch-all buzzword for a huge family of various methods geared either toward achieving better software or toward assessing how good software is. Each of these methods represents a different pipe link, and there are numerous ways of integrating these links that make sense; other combinations are nonsense. For example, putting the system level testing link in front of the processes for checking the requirements for ambiguity is foolish. Not all problems with water (or other products) are self-evident. Management of water quality is like other types of quality management activities and requires risks and opportunities to be managed across the entire water supply network.

Today’s quality practices can be categorized as either process-oriented or product-oriented methods. Process-oriented standards and methods focus on the effectiveness of implementing best practices; CMMI, DO=178, and ISO-9003 all expect a focus on quality across all critical activities, including assurance that this focus is sustained. In contrast, product-oriented verification approaches often presume that you can test quality into a system, rather than assuring that the people, methods, and information involved are all up to the task.

If we return to our analogy of a water system, flow-oriented methods function like connections that are distributed at separate locations along the pipe that allow software representations - whether in a design language or specification format - to be sampled to assess its correctness. The methods that perform assessments would be clustered towards the latter half of the pipeline, since in the earlier phases, code will not be available. These process-oriented methods would attempt to ensure that the original requirements were clean, and that every transformation through the development cycle was not contaminated. Clearly if the water quality is compromised at point A, then at a later point B, it will remain so, unless some filter is employed between A and B. Water doesn’t clean itself any more than incorrect software fixes itself, and water treatment has its limits.

People are prone to making mistakes

You might believe that to assess software quality, all you need are adequately predictive reliability estimation techniques. Most reliability models employ error history information from other situations and use them to forecast future error rates. The problem is that different models can make widely varying reliability projections for the same data, making it impossible to determine which model is the best predictor for a situation. Reliability will be a function of how many errors are injected and detected. Even direct measurement of software quality is less than practical.

Even ignoring all the potential sources of errors in generating test cases, defining test oracles, administrating test execution, and the fidelity differences between development, test, and execution environments, this is unlikely to change anytime soon; the people in the loop are usually the greatest source of defects.

Many of the remaining software assessment approaches focus on metrics. Over one hundred software metrics are in use today, typically focusing on measuring the structural or static properties of the system. The key problem with such metrics is that they do not capture the essence of software. Software is implemented by iterative refinement of mechanisms for transforming inputs into outputs, through a series of prescribed operations. This transformation is the essential characteristic of software development.

Each execution of a program involves a series of state transitions, where each machine state shapes the output. Exactly what effect a particular instruction has on the mapping between program inputs and program outputs is determined by the program’s input space distribution and the program’s control logic and instructions. Structural metrics cannot capture this dynamic aspect of software behavior. This lack of connection with the behavior of the software makes structural metrics especially poor as a predictor of in-service quality performance.

Methods

Research has suggested using fault-injection methods to assess software quality by determining the rate of discovery of these defects. Such techniques purposely inject faults into your software and checks to see whether such injections are detected by other verification methods. However, the types of defects that affect most systems are broad, and the interactions that such defects produce simulates some of these events, and by doing so, it predicts how the software will behave in the future if these anomalous events were to occur.

This entire process of fault-injection relies upon statistics and pseudo-random fault-injection methods. Also, the actual process of instrumenting for the anomalies is quite complex. But the results can be most informative, particularly when you learn that your code doesn’t handle problems quite as well as you thought it would. This information provides an immediate quality improvement opportunity. Interestingly, software fault-injection methods, while they directly assess the behavior (i.e., goodness) of your code, also indirectly assess the goodness of your pipes.

By this, we mean that if software fault-injection methods determine that software is intolerant to either faults within itself or anomalies that can attack the software from external sources (such as human factor errors, failed external hardware, or failed external software), then we learn that the other software development processes failed to build in the necessary water filtering mechanisms. Code must be designed with the proper water filtering systems to ensure that the result from the pipe is clear of defects. If software is incapable of producing the outputs we want under even anomalous circumstances, then we cannot claim that the pipes are clean. And if over time it can be demonstrated (via the results of fault-injection) that the set of pipes you have used repeatedly do result in clean water, then you at least have anecdotal evidence that correlates your processes with the quality of your software.

The quality of your pipes is only one part of what determines the purity of potable water. It is myth to believe differently. Sadly enough, people at the highest levels in governments and corporations have all swallowed this lure, hook, line, and sinker. Quality software does not fail often and never fails in hazardous ways. Software development processes do not define software quality; software behavior does. Behavior is an intrinsic characteristic of software, and it can be viewed without regard to the software’s developmental history.

Before we can have faith in software standards and process models, it must be demonstrated that they have quantifiable effects on the quality of the software produced. Parnas once said: "It seems clear to me that not only is a ‘mature’ process not sufficient, but it may also not even be necessary."

The current popularity of process-oriented assessment techniques is, in part, a reaction to the intractability of performing adequate assessments of software projects. Unfortunately, the relationship between development processes and the attainment of some desired degree of product quality has never been adequately established. This problem is particularly acute for formal methods: forming the required processes does not give quantifiable confidence that the software, when released, will have the required reliability.

In short, testing measures the right thing - product reliability - but it often cannot measure it to the desired precision. Process measurements are more tractable, but they are more effective at managing the flow of work than they are at answering questions like 'how many problems remain'. Some basis must be established to build a data repository from which predictions of uncertain futures can converge with time. And with even more data from such projections (given some underlying framework for the underlying activities and characterization from which several factors such as effort, process yield, and product stability can be analyzed and improved.

1. Skill-based slips and lapses, in which outcomes deviate from intentions because of forgetfulness, misapplication, or lack of awareness during execution.

2. Procedural inadequacies, in which well-meaning processes fail to provide the necessary guidance for consistent execution, often due to descriptive instructions rather than prescriptive ones.

3. Knowledge-based mistakes, in which proper procedures are followed, but the foundational information provided is flawed, leading to failed execution.

To counteract these injection points, systems must include filtering mechanisms that uncover and address such defects. This serves two purposes:

1. To detect mistakes introduced during development.

2. To confirm that outcomes behave as intended.

Recognizing that even our filtering processes are subject to error, we must organize assurance activities with care and discipline. Integration is demanding. It seeks not just technical alignment, but also the removal of embedded defects to an acceptable level. This demands oversight, iteration, and realistic standards of “good enough.”

Refining the approach

Endeavors must have effective evaluation activities which provide adequate filtering to screen out the above sources of errors. We perform these activities for two reasons - to discover errors made during development, and to demonstrate that things work as designers had intended.

Since errors inevitably occur—even within the filtering processes themselves—our methods must be intentionally and rigorously organized. Integration demands meticulous planning and oversight. Its role is to heighten the chances of identifying and eliminating previously introduced defects to an acceptable threshold of quality.

Since verification efforts are prone to error, we complement them with validation activities. These ensure not only that we’re crafting the right solution, but also that we’re executing its construction correctly. However, this layered assurance strategy increases the cost of aligning concurrent design and implementation efforts with intended outcomes—while keeping undesired consequences within acceptable limits.

Integration across teams inevitably encounters a range of challenges: breakdowns in communication, shifting or insufficient definitions of intent, and the added complexity of navigating environmental constraints, unit structures, and the orchestration required by work package providers. These layered obstacles make consistent alignment a formidable but essential task.

Sharpening the focus

In complex systems, no realistic amount of testing can guarantee the absence of all issues. That’s because it's fundamentally impossible to prove a negative. Verification, then, isn’t about asserting perfection. It’s about uncovering defects, isolating symptoms, and qualifying whether a solution meets its intended purpose under practical, constrained conditions.

All assurance efforts are inherently limited by scope, coverage, and relevance—and those limits introduce risk. Unexamined situations may lurk beyond our detection thresholds, poised to trigger failures downstream. Given this, the purpose of verification is not to confirm the absence of error, but to discover enough of the right issues to fuel improvement. It reveals where a system diverges from intention, where gaps persist, and how well outcomes align with stakeholder expectations. In doing so, it builds confidence (not certainty) and sharpens the path from what exists to what was envisioned.

But our ability to discover errors worsens significantly as complexity, novelty, and criticality increases, since such a discovery requires that the error be visible under a specific circumstance. Most requirements describe a subset of these circumstances, but not the whole enchilada. The rest of this footprint is made up of combinations of states, input spaces, and component interactions that typically explodes beyond what can systematically be exercised without consideration of all this information, and the many ways that errors themselves can be injected into a design.

Navigating limitations of evaluations

We simply can't exhaustively test every scenario. All assurance efforts, however diligent, remain inherently constrained in scope, coverage, and relevance. This leaves residual risk: the possibility that unexamined conditions or overlooked combinations may lead the solution to fail in practice.

We must prioritize the time and resources we have available to explore a proper subset of these inputs and states and design our explorations, so we cover the most important and uncharted territory. When testing, we must organize the work by defining a series of testing sequences and conditions which will probe the system's behavior under different operational scenarios and activate threads that adequately traverse our systems' state space as actually built and installed, rather than what was defined, designed, or intended.

But latent errors will inevitably still be out there, like holes in Swiss cheese (Figure 1). Adequate filtering is thus achieved by phases of verification which each have value while also risking overlap in the coverage that can be provided. A defense in depth strategy in this environment can be effective, if the slices don't contain gaps that defects can sneak through.

How small can our footprint be for verification, while still providing the information necessary for assuring that the system will behave properly in service? This has been a classic (and unsolved) problem in testing research for years and was the basis for the introduction of 'structural coverage' into certification guidance for software-intensive systems early in my career. Whether a given set of tests achieves adequate coverage or not depends upon several factors, including the fidelity of the test environment, the stability of the article being tested, and the interactions of the design elements that realistically can be evaluated in isolation vs in situ. Of course, the robustness of the development process itself and the effectiveness of the agents that perform it are equally relevant.

But one will only be able to accurately assess whether a target coverage has been realized after a system enters service, and the real world closes the loop on whether stakeholder expectations for product quality and performance have been realized. Predicting this quality with acceptable confidence before a system enters service implies significant historical data is available on how frequent or significant these discoveries might be, a particularly risky proposition since our lenses are usually rose-colored. However, if we plan for it, it is possible to get scoping indicators about how many defects will need to be removed, and how effective the defect filtering processes of a development and production process are, by monitoring the extent and nature of the defects we discover as we move through stages of development. To understand why this helps, it's important to nail down some terms that provide a semantic framework for the associated concepts.

Some definitions

Clarity demands specificity. Before digging further into strategies or limitations, it's essential to disambiguate what we're talking about. Here's a quick calibration:

• Verification demonstrates whether a product or service meets specified requirements. It answers the question: “Did we build it correctly?” This is typically achieved through inspection, analysis, and testing against documented criteria. It includes evaluating execution fidelity, compliance to specifications, and the degree to which artifacts match expectations.

• Validation demonstrates whether the solution fulfills its intended purpose for stakeholders. It asks: “Did we build the correct thing?” Validation evaluates outcomes in realistic or representative conditions to assess fitness for use, and centers on usefulness, relevance, and whether outcomes align with intent and need.

• Evaluation encompasses broader judgment about worth or adequacy, often factoring in context, tradeoffs, and human interpretation. Assurance is a type of evaluation.

• Qualification is typically concerned with establishing that something meets a defined set of standards or requirements—often to approve readiness or fitness for use under stated conditions.

• Assessments can be informal or formal, qualitative or quantitative, and may apply throughout various life cycle stages to support decision-making, prioritize risks, or guide improvement.

Each of these terms operates at different conceptual levels, some at the level of execution, others upstream of that. Confusing them leads to misalignment in roles, responsibilities, and outcomes. So before orchestrating efforts, we must first reflect on what each term implies, where it applies, and how it's used. If you aren’t already conversant with the language and lifecycle of failures, read this.

Products fail - sometimes often! And despite the best intentions, designs don’t always transform all expectations adequately. Failures can be subtle or spectacular, but they invariably surface limitations in understanding, communication, execution, or context.

Not all failures stem from faulty transformations. Sometimes the ideas themselves were flawed. Other times, real-world conditions challenge even excellent designs. Still others expose gaps in coordination across agents and domains. What matters is not whether failure occurs (it always does) but whether our systems recognize and respond to failure in ways that minimize the impacts of those failures.

We learn from failures. But only when we can see them, attribute them, and synthesize what they reveal. That’s why verification isn’t just a filter. It’s a mechanism for learning, for identifying where and how defects originate, and for clarifying whether those discoveries lead to better decisions next time.

This process isn’t linear or easy. Failures rarely announce themselves with glowing neon signs. Their symptoms may be diffuse, their causes systemic, their solutions elusive. This is why structured verification efforts must do more than tick boxes. They must actively seek feedback, distinguish signal from noise, and help humans interpret what they find.

The other requirements

Beyond correctness and fitness, products must meet a constellation of additional requirements. Some will be explicit, others tacit. These include performance thresholds, safety margins, usability expectations, maintainability, compliance obligations, and others. And they often interact in unexpected ways, creating tension between what’s desirable, feasible, and affordable.

These quality criteria usually include both things that are easy to measure - such as throughput, latency, or capacity - and things that are less easily measured, such as usability or maintainability. Collectively, these are described as 'non-functional' requirements, and while some are amenable to prototyping and rapid iterative refinements, others are much more challenging to evaluate with any meaningful fidelity or regularity. These other requirements aren’t just technical—they’re social, economic, and regulatory. Their definitions may change over time or vary across stakeholders. Some emerge late, in reaction to observed outcomes. Others were implicit all along but only get articulated once something goes wrong.

Efforts to verify these requirements are shaped by how well they’re stated. If a requirement is vague, verification becomes guesswork. If a requirement is over-constrained, it may render competing requirements untenable. Verification must balance precision with flexibility, confirming intent without freezing it.

Critically, these requirements accumulate. They are not resolved in isolation but layered across phases and contexts. Verification must surface their implications, probe their conflicts, and trace their influence. Not just “Did it meet requirement X?” but “What happens when X meets Y under condition Z?” This layering challenges even experienced agents. Requirements propagate through interfaces, architectural decisions, and behavioral expectations. Unacknowledged dependencies distort outcomes. Verification provides a way to interrogate those dependencies, surfacing where assumptions lie and how they’re coupled to judgments in both design, transformation, and execution.

Requirements for methods

Assurance strategies must be tailored to the conditions of the work they’re meant to evaluate. No single method suffices across contexts, so selection must be informed by both the nature of what is being assured and the limitations of how assurance can be performed. To be effective, methods must:

• Align with the intended outcomes Methods must be structured to evaluate against the criteria of success—not just compliance with process, but realization of objectives.

• Accommodate domain-specific constraints Whether constrained by physics, policy, interface characteristics, or resource limitations, assurance must work within the bounds imposed by the endeavor.

• Expose failure modes reliably Effective methods illuminate points of fragility—whether latent defects or emergent behaviors—so that responses can be formulated proactively.

• Support reproducibility and traceability Results should be understandable, repeatable, and traceable back to both the evaluation intent and the observed evidence, enabling oversight and refinement.

• Remain cost-aware and impact-sensitive Assurance isn't free. Methods must balance depth and breadth of evaluation with cost and potential disruption, preserving the integrity of both the work and the teams performing it.

The work products of these methods must satisfy the verification metamodel depicted in Figure 2.

Troubleshooting defects

When defects are discovered, it’s not enough to patch them and move on. Effective troubleshooting seeks to locate the source, trace the pathway of propagation, and understand why detection failed earlier. This is where verification and learning converge.

Every defect is a signal of:

• gaps in integration

• mismatches in interfaces

• unclear requirements

• overlooked assumptions.

These signals only become insight when they’re interpreted. That means troubleshooting must examine:

• Where the defect originated (source)

• How it traveled through the system (propagation)

• Why it escaped notice until now (detection failure)

• What it implies about broader systemic vulnerabilities (scope)

The goal isn’t just to fix the current issue, but to refine the processes that allowed it to happen and remain hidden. Troubleshooting becomes a form of systems thinking by surfacing how behaviors, constraints, and decisions are conducted across domains

Importantly, we must avoid over-fitting fixes to singular symptoms. The temptation is strong to build protective scaffolding around the observed failure. But such scaffolding often decays when context shifts. Instead, verification should seek root causes, not just proximate ones, and ensure corrections harmonize across related concerns.

To prove that a particular defect caused a particular bug, one must determine through observation or analysis that:

• the bug occurs when the defect is present

• the bug does not occur when the defect is removed.

Re-verification then needs to be performed, to provide assurance that no other defects were introduced in the process of removing the original defect.

Done well, troubleshooting strengthens the endeavor itself. It improves coordination. It clarifies ambiguity. It exposes fragilities in assumptions and practices. And it sharpens judgment by turning unwelcome surprises into improved resilience.

The effects of instability

When the elements under scrutiny (be they components, subsystems, processes, or teams) are themselves unstable, assurance becomes a moving target. Instability can shift the meaning of “correctness” midstream, distorting both evaluation results and the validity of claims derived from them.

These instabilities may arise from:

• Rapid iteration or ongoing design evolution Evaluated units change before verification completes, invalidating earlier conclusions or masking unfamiliar problems.

• Poorly defined interfaces or incomplete specifications Ambiguities propagate unpredictably, making it unclear what is being tested or why results matter.

• Behavioral inconsistency under varying conditions Units act differently depending on context, revealing fragility or hidden dependencies.

• Shifting stakeholder expectations or requirements What was “fit for purpose” yesterday may no longer qualify today, rendering prior assurance moot.

Unstable units create an echo chamber of uncertainty: defects are harder to isolate, confidence is harder to build, and alignment with intent becomes tenuous. Assurance efforts lose traction because the terrain keeps shifting. And when instability isn’t acknowledged, misleading conclusions emerge. They can convey false security or premature readiness.

To troubleshoot effectively in this context, verification strategies must accommodate flux. That means incorporating traceability, version control, explicit assumption tracking, and synchronized validation cycles, so evidence stays tethered to the state of what’s being tested, and judgment remains relevant despite change.

The quest for coverage

Assurance efforts must confront an uncomfortable truth: complete coverage is impossible. The sheer number of potential states, inputs, interactions, and failure modes exceeds what can be evaluated exhaustively. This shifts the goal to sufficient coverage, calibrated to risk, complexity, and consequence.

Coverage isn’t about quantity; it’s about quality and relevance. A thousand test cases won’t help if they don’t interrogate the system’s real pressure points. And broad exploration means little without depth in areas where defects breed or propagate. That’s why coverage must be prioritized, weighted, and continually rebalanced as understanding evolves.

To do this well, agents need:

• Explicit scoping criteria What matters most? Which capabilities, behaviors, or interfaces merit scrutiny, given the stakes?

• Risk-informed heuristics Where are defects likely to emerge? Under what conditions would their impact be severe? Prioritization depends on knowing the terrain.

• Traceability mechanisms Coverage must connect back to requirements, intent, and context, so gaps are visible, and evaluations remain meaningful.

• Feedback integration As defects are found (or not), coverage strategies must adapt, shifting focus, deepening inquiry, and refining assumptions.

Without a coverage philosophy, verification becomes reactive and fragmented. With one, it becomes a disciplined search—not for everything, but for enough of the right things. Not proof of flawlessness, but confidence that what was tested matters, and what matters was tested well.

How effective are the evaluations?

Evaluation effectiveness is not binary. It lives in gradients, governed by purpose, context, method, and the kinds of errors being filtered. Even the best assessments may miss what they weren't designed to catch or fail to signal what they did observe with sufficient clarity.

To assess effectiveness, we must ask:

1. What are we trying to detect? Are we screening for:

   1. Design misalignments?

   2. Implementation defects?

   3. Integration inconsistencies?

   4. Deviations from intent or assumptions?

   Different error types—skill-based slips, procedural inadequacies, knowledge-based mistakes—require different filters. Evaluations that are effective against one class may be inert against another.

2. Are the methods aligned? Evaluation activities must mirror the structure of the work they assess:

   1. Procedural reviews work for low-ambiguity, static situations.

   2. Simulation and scenario analysis expose nonlinear or emergent dynamic behavior.

   3. Validation efforts test correspondence between as-built artifacts and intended outcomes.

   An evaluation that isn’t structurally aligned with its target becomes decorative rather than diagnostic.

3. Are feedback loops functioning properly? Effective evaluations generate usable insights. But more importantly:

   1. Are insights acted upon?

   2. Do they lead to refinement in process, practice, or assumptions?

   3. Does the system learn—individually or organizationally?

   Evaluations without feedback loops become static audits, rather than adaptive assurance mechanisms.

4. What is evading detection, and why? An honest assessment of effectiveness requires surfacing:

   1. What errors escaped filtering?

   2. What errors were misclassified or misunderstood?

   3. Why do those breakdowns occur - structurally, behaviorally, or cognitively?

   This is the double-loop learning that reveals limitations not just in outputs, but in mechanisms themselves.

In short, evaluations are effective when they filter what matters, adapt as needed, and clarify limitations. They’re not merely about catching errors—they’re about knowing which errors matter and revealing the shape of our blind spots.

Back of napkin workload sizing example

Estimating workload isn’t merely arithmetic. It’s a form of bounded speculation, mapping assumptions, constraints, and uncertainties into something that can be coordinated across agents and timelines. When done well, it surfaces the latent drivers of risk, misalignment, and rework before they materialize.

Let’s say we’re estimating integration effort for a multi-unit system:

• Known Interfaces Suppose there are fourteen known interfaces, each requiring test harnesses, emulators, or real subsystem coordination.

• Effort per Interface Average effort is estimated at 1.5 person-weeks per interface, including planning, setup, execution, and documentation.

• Integration Complexity Factor Apply a complexity multiplier - say 1.2 - to account for asynchronous dependencies and tooling maturity.

• Coordination Overhead Layer in 20% of effort for cross-team orchestration, churn, and sync-up activities.

Effort Estimation

Total Effort = 14 × 1.5 × 1.2 × (1.2) = 30.24 person-weeks Total Effort

But raw effort tells only part of the story. What matters is how this maps to:

• Schedule feasibility Will this effort meet schedule need dates within available resources?

• Risk exposure Which assumptions are fragile? What if interface definitions change mid-cycle?

• Dependency timing Is interface availability aligned with testing activities?

• Opportunity costs Could some units be initially stubbed instead of waiting for interfacing elements?

Workload estimation is as much about revealing pressure points as projecting timelines. The goal isn’t precision, it’s decision-enabling approximation, tuned to what matters.

Sizing demand

In a large system development of a million lines of code, approximately 50,000 defects will be injected by the team during coding (assuming use of experienced developers); assume half will be detected by tooling. Follow-on reviews, unit checks, standalone tests, integration tests, and system validation tests will each remove additional defects.

If one assumes that each of these verification steps has a yield of 50%, the residual defects remaining after these processes can be projected to be 25,000 * (0.5 / 5), leaving 2,500 defects remaining when the system entered service. Whether this is an acceptable number will entirely be dependent upon their severity and estimated time to repair. If higher quality is needed, possible supplementary actions include adding one or more additional verification steps, increasing the yield per verification stage (through better training or process changes), expanding the time provided to achieve broader verification coverage across stages, or reducing the amount of what must be verified).

Defect isolation

Defect removal time is the time it takes to find and repair a defect. For example, the time it takes to isolate and resolve a problem during validation testing is considerably greater than finding that same defect during a code review. This is principally because during a code review, you are finding the defect directly; when discovering it during validation testing, you must isolate the symptoms and determine the root cause in a much more complex (and typically more expensive) environment.

A defect removal model highlights how yield and defect injection rates interact over time during a component's development and usage, and what the resulting impact on resulting product quality will be.

In this context, an effective strategy for verification and validation will include:

• Establishing an adequate set of activities to be able to accomplish the outcomes which are desired

• Providing appropriate tools and guidance for performing these activities to maximize the yield realized from these actions

• Performing an appropriate level of planning, so that the successive stages of these activities are sufficiently unique in their coverage to achieve the desired quality objectives

• Monitoring the yields being realized by each of these defect removal stages, to ensure that this quality is actually being achieved

• Collecting and analyzing information to accurately predict when artifacts will be sufficiently mature for a particular downstream purpose

These two parameters - defect removal time and yield - will impact what kinds of stages should be designed into verification and validation activities, so that errors are captured earlier (and thus more efficiently), and the desirable quality outcomes are achieved. The degree to which your own verification and validation strategies achieve these goals will have a significant impact on the timely delivery of your desired results!

Verification and validation aren't rituals, they’re lenses which reveal what's working, what’s fragile, and what’s misunderstood. They can transform failure into feedback, and complexity into clarity. But only if we wield them with discipline, purpose, and courage.

Ask yourself:

• Are your assurance strategies tuned to your stakes?

• Are your evaluations filtering what matters?

• Are your processes learning—not just auditing?

Assurance is not just an engineering challenge; it’s a philosophical one. And every defect uncovered is a chance to do better, not just technically, but organizationally and conceptually.