When we work with craftsman to produce products, we rely on their talent, teamwork, and the match of their experience to the jobs to be done. As products grow in complexity, the agents responsible for these solutions must adapt to this complexity. If their performance does not scale as increases in complexity emerge, demand will quickly outpace capacity. This makes it more difficult to locate or develop the required expertise and remain competitive. For technical debt to be retired, and solutions to be?produced?more cost effectively,?systematic, repeatable methods must be adopted that?will produce more reliable outcomes for larger?and more complex efforts.
Structured processes were initially introduced?into such?environments?to mitigate the painful lessons associated?with?more?ad-hoc methods. Yet an emphasis on process is frequently cited as a burden by developers?that constrains them from?pursuing their passion, and getting that which needs to get done, done.
Many have the perception that in-process quality doesn't matter, and that it is?the ends, rather than the means,?that matter. This is like saying that?as long as water appears to be clean, it doesn't matter where it has come from or what has been done with it. Today, countless development and manufacturing standards have been refurbished to attempt to address contaminants discovered after all processing is complete. Too often, organizations have adopted such standards as protectionist legislation intended to keep new competition out, rather than to guarantee higher quality outcomes.
Some definitions
To think through how to do this, some basic definitions are in order. A defect is anything that must be changed in a design or implementation because it detracts from the system's ability to meet stakeholder's needs completely and effectively. Defects can be identified, described, and counted; they are discoverable. The interrelationship between a defect, and its impact (which are manifest through failure, as described below), is complex. Even simple defects can cause large problems, and not every defect can result in a failure (since it can be masked by the architecture, or latent until the required combination of events excites that flaw).
When agents make mistakes that result in defects, these lapses are typically described as injecting defects. The rate at which they do this is measurable, and has been found to be between 50 and 120 defects per thousand lines of code. Half of these defects can be detected by modern tools during coding and unit checkout; an acceptable proportion of the remainder must be removed through down-stream verification and validation activities or risk escaping and affecting operation while the system is in service.
A bug is an observable anomaly which results from a defect and is discovered because an inconsistency with expected behavior is noticed by someone (who hopefully then records this observation so that it can be analyzed and traced to root causes. Bugs typically manifest themselves in the following pattern:
A defect is introduced into a software component, or into a related artifact (requirements, design, or test cases) that is being produced during development. Note that the agents responsible for the component, the artifact, and this verification may be different individuals.
When a path through software logic (either within or across modular units) triggers this defect, either the execution state of the code is no longer valid relative to the intentions of the design, or the artifact will mislead those responsible for implementation or verification
In the case of code defects, this invalid state can itself propagate throughout the code in other ways (as other execution sequences are traversed) to produce other invalid states
One or more of these invalid states may then eventually then result in the bug - the externally visible condition, though this is dependent upon the combination of the execution sequences that are traversed.
A failure may further occur when one or more errors in the system occur, are propagated, and cause the system itself to enter an unsafe, incorrect, or unacceptable operational state. The relationship between defects and failures is thus quite complex, and is highly dependent upon the architecture of the system, and the robustness of its implementation. Frequently, it isn't just one defect or error that leads to a failure, but a combination of several.
What it takes
Development processes require diligence, effort, and appropriate tools in order to shortcut routine operations and help us avoid making slips; unfortunately, both people and tools are fallible. Would you drink water from a glass that was dirty, or drink?from a catch basin that had a visible scum coating the inside of it? Probably not, unless you had no other choice. Yet this insight can be helpful in understanding the importance of having a well-designed process to produce a high-quality product. Believing that dirty?pipes can?produce pure water amounts to accepting the position that flawed tools or oblique?human efforts will?produce good results without accountability. Still, this myth lives on in the minds of both novices, some experts, and particularly bureaucrats.
Quality is difficult to measure?directly in the middle of a value?stream. The operations necessary to transform inputs into acceptable outputs require multiple steps to be performed effectively, across diverse environments, regardless of whether?a business?s products are suitable for human consumption (like drinking water?and?software) or not.?If we stick with this plumbing analogy a bit, we can consider requirements to be similar to the source of a water supply of?drinking water. The value stream?that turns these requirements into useful products is analogous to the links of pipe that fit together to carry water from its source to where it will be consumed. Each link is like the steps in the processes of a value stream. These processes may require defining what kind of?product is needed, developing a work?product, such as a design,?or performing a verification activity to assure that the software complies with its requirements. If the interconnections between these activities?are catch basins, buckets, and funnels, then someone must?make sure the flow accumulating in these?reservoirs is not stagnant.
Eventually something will exit from the pipeline, and with a little luck, it will be a?quality product. The integrity?of the pipeline will?impact the quality of the code. There is no guarantee that even the most superb software development processes will manufacture correct code. Software quality has become a catch-all buzzword for a huge family of various methods geared either toward achieving better software or toward assessing how good software is. Each of these methods represents a different pipe link, and there are various ways of integrating these links that make sense; other combinations are nonsense. For example, putting the system level testing link in front of the processes for checking the requirements for ambiguity is foolish.?Not all problems with water (or other products) are self-evident. Management of water quality?is like other types of quality management activities, and manage risks and opportunities across the entire water supply network.
Today?s software quality methods can be generally labeled as either process-oriented or product-oriented. Process-oriented standards are focused on achieving a quality outcomes rather than guaranteeing a particular flow rate. Process-oriented standards focus on the effectiveness of certain behaviors compared to best practices; CMMI, DO178, and ISO-9003 all expect a focus on quality across all critical activities, including assurance that such focus is maintained as products come and go. In contrast, product-oriented verification approaches often presume that you can test quality into a system, rather than assure that the people, methods, and information involved are all up to the task.
If we return to our analogy of a water system, product-oriented methods function like connections that are distributed at different locations along the pipe that allow the developing software, whether in a design language or specification format, to be sampled in order to assess its correctness. Those product-oriented methods that are specifically for software assessment would be clustered near the exit from the pipeline, since in the earlier phases, code will not be available. The process-oriented methods would attempt to ensure that the original requirements were clean, and that every transformation through the development cycle did not inject dirt. Clearly if the water is dirty at point A, then at a later point B, it will still be that dirty and possibly even dirtier, unless some filter is employed between A and B. Dirty water doesn?t clean itself any more than incorrect software fixes itself; water treatment?has its limits.
You might believe that to assess software quality, all you need are adequately predictive reliability estimation techniques. Since it is difficult to predict the reliability of a piece of software, most software reliability models employ error history information from other situations and use them to forecast future error rates. The problem is that different error-history based reliability models often produce different reliability projections for the same data, making it impossible to determine which particular model is the best predictor for a specific system. The state-of-the-practice in software reliability assessment is simply not up to the task, especially since reliability will be a function of how many errors are injected and detected. Even direct measurement of software quality is less than practical. Even ignoring all the potential sources of errors in generating test cases, defining test oracles, administrating test execution, and the fidelity differences between development, test, and execution environments, this is unlikely to change anytime soon; the people in the loop are usually the greatest source of defects.
Many of the remaining software assessment approaches focus on metrics. Over 100 software metrics are in widespread use today, and usually focus on measuring the structural or static properties of the system. The key problem with structural metrics is that they do not capture the essence of software. Software is implemented by iterative refinement of the mechanisms and implementation by which an input is transformed into an output, through a series of mechanical operations. This transformation is the essential characteristic of software development. Each execution of a program involves a series of state transitions, where the machine state contains the output. Exactly what effect a particular instruction has on the mapping between program inputs and program outputs is determined by the program?s input distribution and the program?s instructions. Structural metrics cannot capture this dynamic aspect of software behavior. This lack of connection with the behavior of the software makes structural metrics especially poor as software quality assessors.
Research has suggested using fault-injection methods to assess software quality by determining the rate of discovery of these defects. Such techniques purposely inject faults into your software, and checks to see whether such injections are detected by other verification methods. However, the types of defects that affect most systems are broad, and the interactions that such defects produce simulates some of these events, and by doing so, it predicts how the software will behave in the future if these anomalous events were to occur.
This entire process of fault-injection relies upon statistics and pseduo-random fault-injection methods. Also, the actual process of instrumenting for the anomalies is quite complex. But the results can be most informative, particularly when you learn that your code doesn?t handle problems quite as well as you thought it would. This information provides an immediate quality improvement opportunity. Interestingly, software fault-injection methods, while they directly assess the behavior (i.e., goodness) of your code, also indirectly assesses the goodness of your pipes. By this, we mean that if software fault-injection methods determine that software is intolerant to either faults within itself or anomalies that can attack the software from external sources (such as human factor errors, failed external hardware, or failed external software), then we learn that the other software development processes failed to build in the necessary water filtering mechanisms. Code must be designed with the proper water filtering systems to ensure that the end result from the pipe is crystal-clear water. If software is incapable of producing the outputs we want under even anomalous circumstances, then we cannot claim that the pipes are clean. And if over time it can be demonstrated (via the results of fault-injection) that the set of pipes you have used over and over again do result in clean water, then you at least have anecdotal evidence that correlates your processes with the quality of your software.
The quality of your pipes is only one part of what determines the purity of potable water. It is myth to believe differently. Sadly enough, persons at the highest levels in governments and corporations have all swallowed this lure, hook, line, and sinker. Quality software does not fail often, and never fails in hazardous ways. Software development processes do not define software quality; software behavior does. Behavior is an intrinsic characteristic of software, and it can be viewed without regard to the software?s developmental history. Before we can have faith in software standards and process models, it must be demonstrated that they have a quantifiable relationship to the behavior of the software produced according to them. Parnas once said: "It seems clear to me that not only is a ?mature? process not sufficient, it may not even be necessary."
The current popularity of process-oriented assessment techniques is, in part, a reaction to the intractability of performing adequate assessments of software projects.? Unfortunately, the relationship between development processes and the attainment of some desired degree of product quality has never been adequately established. This problem is particularly acute for formal methods: forming the required processes does not give a quantifiable confidence that the software, when released, will have the required reliability. In short, testing measures the right thing - product reliability - but it often cannot measure it to the desired precision. Process measurements are more tractable, but they are more effective at managing the flow of work than they are at answering questions like 'how many problems remain'. Ultimately, some basis must be established to built a?data repository from which predictions of uncertain futures can converge with time. And with even more data from such projections (given some underlying framework for the underlying activities and characterization from which various factors such as effort, process yield, and product stability can be analyzed and improved.