Standards which stand the test of time
DO-178 is the primary document used by aviation certification authorities throughout the world. The document is used to assess the adequacy of safety-critical practices used in developing software for commercial aircraft. Certification is a complex public good, and is described in the latest version of DO-178 as follows:
Legal recognition by the certification authority that a product, service, organization or person complies with the requirements. Such certification comprises the activity of technically checking the product, service, organization or person and the formal recognition of compliance with the applicable requirements by the issue of a certificate, license, approval or other documents as required by national laws and procedures. In particular, certification of a product involves: (a) the process of assessing the design of a product to ensure that it complies with a set of standards applicable to that type of product so as to demonstrate an acceptable level of safety; (b) the process of assessing an individual product to ensure that it conforms with the certified type design; (c) the issuance of a certificate required by national laws to declare that compliance or conformity has been found with standards in accordance with items (a) or (b) above.
There are many unique challenges associated with the development of such safety-critical software. The projects which must produce this kind of product typically have to deal with issues such as constrained resources, novel system architectures, and real-time performance constraints; these amplify the already difficult challenge of successfully developing software within fixed time allocations and to very demanding requirements.
The introduction of regulatory oversight and conditions into this environment can either help or hinder achieving these goals, depending upon how effectively and carefully this oversight is implemented. If insufficient attention is given to key details during project execution, or if there is a lack of understanding of the underlying requirements necessary for implementation, the risk to the resulting system development activities and products will be high.
Purpose
The purpose of DO-178's guidance is to mitigate these risks by providing sufficient information to successfully plan and implement development activities so that certification can be achieved. When experienced software certification professionals and airframe manufacturing applicants both adopt common guidance, it helps them make consistent determinations of compliance regarding design, development, and operational practices for aircraft and related ground systems.
DO-178 is neither a software development process nor a prescriptive standard for such a process. Instead, the document serves to structure and guide negotiations between applicants and certification authorities on the activities and deliverables which must be accomplished during product development and support activities. The guidance strives to balance the need for tangible material that provides a consistent basis for these determinations against the range of existing airborne software and company processes that are used in actual practice across the industry.
Interfaces
As a result, while the document provides key compliance criteria, it does not provide specific, useful information on how to apply this information on development programs, in a cookbook, 'how-to' manner. The document will thus only be effective when correctly interpreted and applied by experienced professionals, as they prepare, approve or recommend approval of technical data for submission to certification authorities. This style is consistent with the mentoring and selection process for Designated Engineering Representatives across the industry, and fundamental to Organization Designation Authorizations now in place.
The complexity and extent of the compliance data required by DO-178 depend upon the characteristics of the system/software, associated development practices, and the interpretation of DO-178 guidance, especially when it must be applied to new technology and situations in which there is little or no precedent available to draw from. Two different Airworthiness Representatives, working from the document, may still reach different decisions about the acceptability of a particular artifact, because of their experience, preferences, and situation; no standard can or should eliminate technical judgments by qualified agents.
Concepts of operation
Like any standard, each version of DO-178 had good points and bad points (and some versions even contain a few errors). However, careful consideration of its contents, combined with solid engineering judgment, is intended to produce better and safer software. The material is not always popular or easy to implement, but neither is the challenge of assuring the safety of flight. Insights about how the guidance is applied in the real world are also fed back to accomplish continuous improvement of the standard itself.
As an example of how criteria are invoked, some believe that the enhanced rigor and completeness of verification evidence required by DO-178 represents a significant cost burden on the avionics industry. Representatives have complained that producing this evidence requires an inordinate amount of effort, and question the value resulting from this effort. Such views must always be considered in the context of the standard's mission. Since the original emphasis of the DO-178 document has always been safety rather than cost-effectiveness, companies will naturally have widely varying approaches to implementing DO-178, and some are clearly more cost-effective than others.
Call to action
The initial version of the document was published in 1982, just prior to certification of the first models of Boeing's 757 and 767 aircraft. When the first revision, DO-178A, was published only 3 years later, the avionics industry had just completed a major transition from analog to digital systems, in which software played a much larger and more prominent safety-critical role. As a result of this transition, many more companies were subject to certification oversight and compliance determinations. During this initial use, an unacceptable portion of these companies struggled to successfully certify their systems, because these versions of DO-178 were not sufficiently detailed to provide them with sufficient implementation guidance.
Because of these shortcomings, the industry quickly came to a conclusion that still another version of DO-178 was needed. The new version was expected to incorporate experience from these initial applications of prior versions of the guidance and to address a number of questions that were being raised as a result of emerging technical issues and proposed new approaches. In some situations, these questions required new rules to be established, such as in identifying mechanisms for certifying reusable software components, qualifying tools, assessing object-oriented design impacts, and dealing with inoperative code in operational systems.
Emerging drivers
To pick just one of these examples, DO-178 activities are expected to produce rigorous and complete verification and traceability evidence, and the efforts required to produce this information can be quite substantial. As the complexity of software has increased, the cost to produce this verification information has also grown proportionally. While tools have always been used in many situations during development, emerging tools were envisioned to play an increasing role far beyond what had previously had been attempted. This reliance upon such tools, potentially with limited reviews of their results and artifacts, motivated the industry to consider a more explicit definition of tool qualification criteria, and to integrate those criteria into the development practices and descriptions of the rest of the document.
The DO-178B revision was also expected to align its concepts and terminology with emerging guidance for systems development which was concurrently being developed. This guidance, documented in DO-254, provides criteria for design assurance of airborne electronic hardware and describes the expected information required throughout project conception, planning, design, implementation, testing, and validation. For example, through joint agreements between the two standards efforts, it was agreed that the terminology used to describe the criticality of safety considerations needed to change from "critical, essential, and nonessential" to "catastrophic, hazardous/severe-major, major, minor, and no effect". To reflect these changes, the software levels which had originally been described in DO-178A as "Level 1, Level 2, and Level 3" would also need to be changed to "Level A" through "Level E" in DO-178B guidance. This change enabled correspondence between system and software levels to be made when accompanied by appropriate system design and implementation techniques that were used during development.
More significantly, a means of clearly defining and communicating the effect of these changes needed to be established for these corresponding requirements as these levels changed. For example, with respect to the types of verification test coverage required for Levels C, D, and E software, only statement coverage needs to be achieved, but for Level B software, decision coverage is required, and for Level A software, each part of a condition must be demonstrated to correctly affect the outcome through the program’s logic. The standards body needed to determine how these differences would be defined, and the document had to delineate them in a way that was clear, concise, and appropriate to the situation.
After consideration of all of these changes, it became apparent that a complete rewrite of DO-178A was required, despite the fact that this would be the third revision developed for this community in only 5 years.
Because of concerns about the direction which this updated version could take, hundreds of participants from across industry were involved in authoring, reviewing, and integrating the resulting DO-178B content. Organizing their efforts required developing mechanisms so that key issues could be surfaced and analyzed, alternatives weighted, and specific text for proposed guidance could be proposed so that each community of interest could form, storm, and norm there way towards consensus viewpoints.
There was significant pressure to 'get this one right'. In light of the volatility of prior efforts. With the benefit of hindsight, it is worthwhile reviewing the keys of success to this effort, which collectively enabled us to successfully manage the scale and nature of participation for the overall standardization, and to improve the quality and usability of the resulting guidance, and its acceptance by the affected stakeholders:
There was skin in the game for the participants
The development of aircraft systems and the software embedded in these systems rely heavily upon a 'safety culture' that actively focuses on producing software with dramatically fewer safety-related defects than traditional practices would normally produce. In such cultures, with lives and company reputations at stake, the opinions of practicing colleagues tend to be much more highly valued than outsiders. The engineers involved in these development efforts must pay meticulous attention to detail, and act deliberately as safety-related requirements and issues are identified. As they do this, other needs like cost-effectiveness, schedule predictability, or adoption of the latest technology innovations, while important to the business, are understandably less important than the focus of DO-178B, which is on assuring the development of software which will operate in life-critical systems.
In this situation, the committee participants recognized that the guidance which would be produced by committee activities would be of primary importance when performing their duties in their 'day jobs' but would not be the only guidance they would need to deal with. While DO178 is described as guidance throughout the document, this is because regulatory authorities describe it as an acceptable means of determining compliance for aircraft certification, without closing the door to other means. In all other ways, however, this document provides the minimum acceptable provisions (i.e. the standards) which must be taken into account in planning and implementing software-based development activities for products that are to receive regulatory approval.
For many other industry standards, such a 'mandate for use' is either not in place or is not achievable because of the way the guidance has been expressed or invoked. Further, compliance in these other situations often does not need to be demonstrated in any formal sense (though groups may still claim they are compliant, without saying by how much), or to whom and how that demonstration has been secured and recorded.
Decisions about the content of DO-178B in this context were made by striving for consensus by all participants. Consensus does not mean agreement, but rather acceptance, and requires a give and take by all participants to find common ground and jointly strengthen that position. Achieving this consensus across hundreds of interrelated issues required many meetings to realize but paid off in the quality of the resulting document and the endorsement by the broader community. When consensus could not be achieved, minority decisions were documented and were accepted by the Special Committee chairs, but this path was only found necessary in extremely limited cases, and all these concerns were resolved by final publication.
Community engagement
Such consensus-based standards may be biased towards guidance which retains the ability of industry members to refine (rather than replace) their current practices and require judgment by experienced individuals to make detailed compliance determinations. To reinforce, although a standardized format for documentation might be preferred by regulatory groups, any such format selection would likely favor some companies or groups over others. In a team of competing industry participants, any such specific recommendation would favor some over others and thus would fail to achieve the necessary consensus buy-in. The relevance of the resulting guidance with respect to the diverse implementation approaches used over this period is a direct consequence of the team environment which was fostered as the standard was developed and ratified.
Over time, this requirement for compliance demonstration has resulted in forming a community of practitioners from many different perspectives. These practitioners can come together and have meaningful discussions about development practice trade-offs and their consequences. Such trade-offs are a part of the life of every system development. At the end of the day, DO-178 and its processes don't certify safe systems; this community does. The importance of their engagement in this reflection and self-regulation cannot be over-emphasized, and such involvement by practitioners has often been missing from other efforts to crafting a body of knowledge?for other occupations or industry segments.
Terms of reference were established to scope and provide accountability for the effort
Terms of reference were established for the DO-178B Special Committee by the RTCA board to focus and stabilize the efforts of the working groups. These objectives called the group to:
Examine existing industry and government standards and consider for possible adoption or reference, where relevant
Assess the adequacy of existing software levels and the associated nature and degree of analysis, verification, test and assurance activities. The revised process criteria should be structured to support objective compliance determination
Examine the criteria for tools to be used for certification credit - for example, tools for software development, configuration management, or verification
Examine the certification criteria for previously developed software, off-the-shelf software, databases, and user-modifiable software for the system to be certified
Examine the certification criteria for architectural and methodological strategies used to reduce the software level or to provide verification coverage, for example, partitioning and dissimilar software
Examine configuration control guidelines, quality assurance guidelines, and identification conventions, and their compatibility with existing certification authority requirements for type certification, in-service modifications and equipment removals
Consider the impact of new technology, such as modular architecture, data loading, packaging, and memory technology
Examine the need, content, and delivery requirements of all documents, with special emphasis on the Software Accomplishment Summary
Define and consider the interfaces between the software and system life cycles
Review the criteria associated with making pre- and post-certification changes to a system
Consider the impact of evolutionary development and other alternative life cycles to the model implied by DO-178A
This direction served as the charter for the team's activities; these terms of reference were reviewed, revised, and accepted by the entire Special Committee before being finalized. This helped to assure that the wording of this charge was understood and accepted by the members who would be implementing this mandate.
Working group responsibilities for developing content were defined
When dealing with hundreds of participants from many different sectors - regulatory agencies, airframe manufacturers, and suppliers - and many perspectives - systems engineering, quality assurance, software engineering, and management - it was necessary to organize committee activities into parallel groups. A working group structure was established so that participants could self-select where they could best contribute. Working group chairs were then selected who were committed to working together to resolve cross-group boundary issues, and who could provide the necessary leadership and guidance to help drive issues to closure within each of these sub-groups. I was the leader of the working group responsible for change management, coordination of key concepts, and integration of guidance.
Governance to manage information exchanges and decision-making was established
Governance is often an afterthought when groups are themselves developing?new governance. However, because of the number of people involved in developing and reviewing DO-178B, effective governance practices were essential to establishing trust and managing the evolution of the material being developed. A table of contents was established early on to describe the intended architecture of the document and provide a basis for assigning responsibilities to the working groups. A template for working papers was also established as a means to identify, track, and communicate progress on critical technical decisions within each working group over time, in a trade study like environment. Key concepts and features that affected multiple sections of the document were then identified, to ensure that agreed-to approaches for these concepts were developed before tackling other related issues. For example, the group wanted to avoid specifying any particular structure for required artifacts and documents that were to be produced during a DO-178B-compliant development effort, and instead of using these traditional artifact descriptors, made a conscious commitment to describe information in abstract containers that could be implemented in a variety of ways; this decision rippled through many other sections of the material that was developed. Finally, a means of identifying and tracking issues for existing content was established. This mechanism satisfied multiple purposes for the teams' authoring efforts and served as problem reports, change requests, and requests for the resulting information. The communications, tracking, and disposition of these issues were provided centrally in support of all working groups as a service for all participants.
A terminology foundation was documented and maintained
Different groups, and individuals from different backgrounds, often use this different wording in developing content they are familiar with, even though they had the same underlying meaning in mind. These same groups, in somewhat different circumstances, may also use the same wording even though they intended their wording to mean different things. Such communal differences needed to be resolved in order for the resulting guidance which they produced to be coherent.
As a classic example, consider the terms 'verification' and 'validation', which traditionally have many different definitions and interpretations. In DO-178B, verification is defined to be the 'evaluation of the results of a process to ensure correctness and consistency with respect to the inputs and standards provided to that process'. In DO-178B, verification is one of the integral processes (configuration management, quality assurance, and certification liaison), which means it cuts across all the development processes, rather than being a serial activity which begins after a predecessor activity. Validation is defined as 'the process of determining that the requirements are the correct requirements and that they are complete'. Since validation is considered a system-level activity, it is not explored further in DO-178B. Definitions such as these form a solid foundation which is used to tie the other sections of DO-178B together but need to be established upfront so that the content built upon these definitions could be integrated together with a minimum of rework.
Such verification is a part of the checks and balances which DO-178B provides to protect against mistakes. Additional protection is required when this verification must be performed independently, which is defined in DO-178B as follows:
Separation of responsibilities which ensures the accomplishment of objective evaluation. (1) For software verification process activities, independence is achieved when the verification activity is performed by a person(s) other than the developer of the item being verified, and a tool(s) may be used to achieve an equivalence to the human verification activity. (2) For the software quality assurance process, independence also includes the authority to ensure corrective action.
Once definitions such as the above were established, key concepts could then be built from this foundation. As an example, consider the document's treatment of software requirements, which are described as "a description of what is to be produced by the software given the inputs and constraints". They take two forms in DO-178B: 'high-level' and 'low-level' requirements. High-level requirements are defined as "software requirements developed from analysis of system requirements, safety-related requirements, and system architecture", and low-level requirements are defined as "software requirements from which source code can be directly implemented without further information". This definition can obviously be interpreted differently (and thus require further elaboration of low-level requirements) when applied to a new-hire and an experienced engineer, which could drive up costs unnecessarily if this elaboration proves unnecessary once the coding responsibility is assigned. The importance of independence in verification is also highlighted in this situation to ensure that the low-level requirements are taken to the required level.
Development process objectives and interfaces were baselined
Compliance with DO-178B is intended to establish confidence in the activities used to develop safety-critical software and assuring that these activities are producing information suitable for certifying the systems in which the software is embedded within. DO-178B's life-cycle processes are not intended to prescribe or limit the structure of the actual development processes which are put in place by development organizations; instead, the processes described in DO-178B are a descriptive vehicle for communicating expectations about the activities actually performed during development. While these actual activities may take many different forms, DO-178B defines its own set of software life-cycle processes so the actual activities can be traced to the DO-178B processes.
A set of objectives for the DO-178B lifecycle processes are also defined. These objectives define the 'intent' behind these processes and thus describe the value which each of these processes is expected to produce, in a manner like how a judiciary may consider legislative intent in determining their rulings. The objective evidence necessary for demonstrating that these objectives have been accomplished is also described. For example, testing objectives are defined as showing that requirements are fully satisfied, and demonstrating 'with a high degree of confidence that errors which could lead to unacceptable failure conditions... have been removed'. There are obviously many different approaches that can be employed to achieve such objectives, but the proof of their pudding is whether they satisfy these objectives or not. DO-178B objectives also help verify the correct implementation of safety-related requirements that flow from the system safety assessment.
Compliance data is intended to be produced as a natural product of these activities, but unlike many other standards, DO-178B does not prescribe how these activities should be structured. Interpretation and judgment need to be applied in determining the activities and deliverables appropriate to each situation in order to satisfy both DO-178B process objectives and criteria, as well as any business objectives that must also be satisfied (which are outside of the scope of DO-178B).
An evolutionary development plan for the document was outlined
Concepts in any standard need to be developed and communicated in an organized and sequenced fashion so progressive elements can be used to develop the reader's understanding as the document unfolds. A top-level document structure was derived from a consensus view of how certification programs would be described. An outline of the overall document was established, and target page counts were established for different sections, with the overall objective of keeping the length of the document to approximately 75 pages. The thinking was that more writing does not generally lead to clearer writing.
This content for this structure had to be developed incrementally by the responsible workgroups, and this material thus evolved over many meetings in which the working group members held the majority of their team's interactions. As content for the document progressed through position paper development to working-group agreed-to proposals for content and presented that content to plenary, it was apparent that the underlying concepts often did not yet 'stand-alone', and had to be explained by the working group chairs to the rest of the plenary members.
For example, a key change was made in DO-178B in how requirements for software verification were to be defined, with a greater emphasis on requirements-based testing than in DO-178A. This change was intended to discourage the practice of focusing software testing primarily to achieve structural (i.e., White-box testing) coverage, at the expense of fully testing functional and performance requirements. Requirements-based testing - augmented by a determination of the structure coverage achieved by these functional tests - is a more cost-effective and meaningful way to conduct such verification testing.
The emphasis on requirements-based testing resulted in a new and significant focus area in DO-178B, which expects consistency between requirements, code, and the tests that verify that the code satisfies the requirements. Traceability is often used to provide evidence of this consistency and completeness, and such traceability is required to be thorough and bidirectional, demonstrating that requirements trace forward to code and tests and satisfy requirements, and trace backward from tests to code and to the requirements for which the tests were created. At the higher levels of safety and criticality, DO-178B requires that this traceability evidence show 100% structural code coverage. These concepts were each developed within different working groups and were only able to achieve consensus after multiple iterations in the document test - to introduce the idea, integrate its expression, and mature its elaboration. A roadmap of these concept developments and refinements over time (which initially were communicated in the form of working group papers) was helpful to begin developing realistic projections for how long DO-178B would actually take to complete.
A repository was established for sharing information and status
An accessible means of exchanging and providing access to information was put in place to support the committee activities. This mechanism helped improve configuration control of document elements, provided protected access to working groups for the working group papers that were not ready for broader review, and helped build a community of practice to identify concerns and leverage community knowledge in trying to address them. The repository that was established was funded in a creative manner, with software licenses paid by one airframe manufacturer, the hardware paid by another, and the server and associated operations provided by a university that was actively involved in the committee activities. This repository has been so useful that has been upgraded many times and remains in use today.
An editorial team was established to integrate content into a coherent product
While the content of the document was developed by many individuals, it was important for the resulting integrated content to be written as if it had been the work of a single author. Since the document was to be international (and would thus need to be translated into other languages), care needed to be taken to assure that the language and structure of the document were amenable to this translation and that it could be accomplished without changes in the underlying meaning. As an example, the process of progressively refining requirements was described by some authors as 'decomposing requirements', but such terminology, while often used within the United States, would not translate well into French (essentially becoming 'rotting requirements'. Finally, the team needed to establish trust with the original content providers, who wanted the intent of their original writing to be preserved. Over time, what this meant was that responsibilities for the document sections transferred from individual working groups to the editorial team for final integration.
A formalized review process was employed to assure quality criteria were realized
The editorial team employed a structured review process, including a specific checklist designed for such reviews. As these reviews identified issues with the document, the working groups were initially responsible for addressing these issues and determining if the resolution required a change in a technical concept, or how it was explained. As the document entered its final form, change authority was delegated to a smaller number of individuals, to make sure the tone and style were consistent, as the number of ‘must fix’ open issues was driven to zero.
Results
Work on position papers for the document began immediately after the first meeting, using the working group structure that was established in plenary sessions. The first drafts of DO-178B began to be produced in 1989. Work continued through quarterly meetings, with the intervening publication of document updates, exchanges of issues and working group papers, and progressive refinement and convergence on the agreed-to guidance. The final release was adopted in 1992 after the responsibility for the document text was transferred to an editorial board. This team, with representatives from other working groups, conducted intensive sessions to thoroughly review the document, reconcile outstanding issues, and prepare the document for publication. The resulting product has achieved wide acceptance and remains in active use after over thirty years on all new and derivative commercial aircraft developments around the world. The document's usability, clarity, and compatibility with other standards have been well received.
This is one of the proudest accomplishments in my career. I served as the document’s architect, led the editorial group’s efforts to integrate the text produced by all the working groups, and massage their drafts into a holistic document. I received this much-appreciated note (Figure 2) from the overall cochair of the effort acknowledging my contributions:
A new revision, DO-178-C, was initiated in March 2005 and was completed in 2013. Refinements clarified and refined the definitions and boundaries between the key DO-178B concepts of high-level requirements, low-level requirements, and derived requirements, and incorporate the exit/entry criteria between systems requirements and system design (described in ARP-4754A). Since this material has now been widely used in practice for over thirty years, the above practices collectively offer advantages over less formalized approaches that have been used in other standards efforts but are applied in practice very infrequently.
Practices are only effective when used by the right people and the leaders and contributors to DO-178B can remain proud of their contributions to the practice of software engineering in safety-related systems. The above practices are offered in the hope that this evolution can produce a similarly enduring legacy in future work.