A pilgrim's search for elegant solutions
Learning to embrace uncertainty and grind through pathways to success
Booting up
Boeing hired me fresh out of graduate school in 1978, armed with imparted knowledge (but little practical experience) in computer science theory: compiler and operating system design, data structures and algorithm performance, and survival strategies for batch programming environments. I worked full time from my sophomore year throughout college and graduate school in the university’s medical research laboratories to set up and maintain instrumentation, assist in research studies, and perform statistical analysis on the results, co-authoring several publications.123. While I had many offers from most of the computer companies of that time, Boeing had a huge attraction advantage for me given its headquarters in the beautiful outdoors of the Pacific Northwest.
My specialization emerges
My first Boeing assignments introduced me to the world of real-time data processing using minicomputers - now relics in museums - but at the time were signaling a major shift away from computing in centralized data centers onto dedicated devices for specialized applications. This shift also enabled broader applications to the specialty area of real-time data processing, an emphasis that would dominate my career.
Upon my arrival, Boeing was actively managing the concurrent development of two new aircraft, the 757 and 767. The cancellation of Boeing’s SST in the early 1970’s - which led to the infamous Boeing Bust - was finally behind the company, and it was again the dominant force in commercial aviation, with a formidable reputation in the industry for its innovation and engineering excellence. As the company has often been known to do, it bet its entire future on developing these two new aircraft to take maximum advantage of the emergence of digital avionics. By automating the monitoring and operations performed by the flight engineer, this innovation enabled reductions in crew which had enormous economic benefits to airlines, a bellwether for the effects of artificial intelligence in many areas today.
Neither of these new aircraft would be allowed to leave the ground even for test flights until the structural integrity of the aircraft designs had been adequately demonstrated and bought off by certification authorities. While similar tests had been performed on prior test programs - most recently for the 737 (which entered service 11 years earlier) and 747 (which entered service 9 years earlier), the retained knowledge from these efforts was limited as 60% of the company’s workforce had been laid off earlier in the decade. These ‘boom or bust’ cycles were a part of the Boeing culture and contributed to the strength of unionization of most manufacturing and engineering employees in the Puget Sound region.
One of my earliest assignments was to write software for a new array processor which the hardware test group had recently procured - at considerable expense - to support this structures testing. Interestingly, in this era, hardware was expensive, and software was comparatively affordable. The resulting integrated system needed to collect about one thousand samples of raw data in real time at ten millisecond data rates and convert those samples into engineering units, which were necessary to calculate stress and strain in support of full-scale aircraft fatigue testing, a 24x7 operation. These test systems required continual monitoring of these critical parameters to ensure values were within pre-established safety margins and to provide accurate data necessary for aircraft verification. Any failures in this process could severely compromise both programs, with few options for recovery. The endeavor had considerable skin in this game, as this testing was in the critical path for completing both the 757 and 767 aircraft’s development on time.
Escalation of responsibilities
My group was growing as it supported these efforts; after two years, I was promoted to lead engineer for a team of six software engineers, which quickly built out multiple real-time data collection and analysis systems to support these testing efforts. The systems needed to be real-time as continuous monitoring was necessary to protect against applied loads creating hazards for the test operations team, to protect against damage to the test article itself, and to assure that adequate information would be captured for subsequent redesign should the articles fail during testing, Specialized hardware was often required, including a hybrid analog-digital computer necessary for testing the first commercial aircraft anti-skid brake systems in our laboratories.
In parallel, we were asked to to implement a multi-vendor communications network to exchange data across Boeing’s Commercial Airplane division’s many test facilities. We had few tools available in those days besides the compilers, editors, and operating systems that were provided by the computing manufacturers. My team worked extremely well together, and we all experienced the joy that results from collaborative success as the team was able to overcome many challenges.
Despite this added workload, I continued to be a technical contributor on the team, along with mentoring, overseeing, and integrating the efforts of other team members and interfacing with management. Our array processor was delivered and required crafting custom device drivers and interfaces for the data recording and analysis software which was being built by my team in cooperation with our customers. We wrote this systems software in assembly language, while our applications were written in the programming language of choice in that day, Fortran.
My development pattern emerged
My take on this new device was that no one really understood how this array processor would work (or even if it would be adequate to the task) prior to the purchase. Critical elements like the bandwidth of the buses involved, the write performance of the disks, and the time budget required for software functions to perform the necessary calculations had not been considered; our mindset was to “buy the best and caulk the rest”. My plan was to explore, prototype, experiment, evaluate, learn, and iterate. Many late nights were involved, debugging code generation errors from immature compilers, troubleshooting problematic hardware, and ensuring my own code was adequate to the demands of the real-time environment.
During this time, I also introduced a new customer communications construct into our group which we ended up calling a Project Review Board. This PRB involved key customer subject matter experts in weekly meetings to discuss near-term plans and consider options in play at that time for the development team. Each month I also had a one-on-one ‘accountability review’ with the broader organization’s chief engineer to discuss the progress our team had made over the previous month. These habits of iterative development, team engagement, management coordination, and customer collaboration would form the essence of modern agile software development practices and serve me well for the rest of my career.
Build the tool to build the system
Later in my career, Boeing decided they no longer wanted to keep copies of personnel files in the company, and they sent mine to my home, which included a listing of the code which I wrote during this period, and which I’ve been able to review over 4 decades later (figures 1 and 2).
The critical code had to be written in assembly language (something few programmers use today). My underlying design objective was to hide secrets from customers and application programmers about where data was being kept in memory and in the array processor, with the goal of providing flexibility to enable those decisions to be ‘tunable’ to optimize performance for different situations. I also wanted to utilize constructs that would be familiar to end users. The design documentation was embedded in the code, which was my practice; a sample of this is provided in Figure 2 below and reflects a ‘hack’ exploiting Fortran’s statement functions, which overload array syntax. Hence, applications access data via traditional array elements, but at the machine level accessed the array processor elements via the device driver, which also maintained a cache of recently used values for optimum performance. This allowed us to capture and compute 1000 channels of data at 10 millisecond rates, which was considered the state of the art in that time. Looking back, I’m grateful it worked, because if it hadn’t, my career would have taken a different path entirely. By the time the next generation of aircraft required such testing, the performance of computers had advanced to the point that these specialized array processors were no longer required, and decisions began to be made to implement as much functionality as possible in software.
Programming in assembly language can be challenging because that level of coding provides little structure for decision logic or program flow. As a team, we decided to adopt a convention of peer reviewing our designs once they were sketched out using a standardized pseudocode syntax. I worked with another engineer to build a preprocessor for this pseudocode which could check for errors and format these designs according to this structure and then migrate that pseudocode into comments in the assembly language code, continuing to prettyprint the text with indents and line wraps to promote readability according to the control structure of the pseudocode. I then built a series of assembly language macros which matched that same control structure and which autogenerated the assembly jumps necessary to implement that structure. Figure 3 below provides a sample code fragment from what that looked like:
Payoffs
The processor and macros ended up being widely used on the platforms we were using at the time (Data General hardware), which was a gratifying experience.
The state of technology has advanced to the point today where such techniques are rarely visible to programmers; the capabilities of operating systems and tool chains are really the foundational miracles of our world. However, there are always space and time limitations somewhere, and good engineers always find ways to battle through such constraints and produce solutions that stand the tests of time. And that is a most gratifying accomplishment.