A pilgrim's search for elegant solutions
Learning to embrace uncertainty and grind through pathways to success
Boeing hired me fresh out of graduate school in 1978, armed with imparted knowledge (but little practical experience) in computer science theory: compiler and operating system design, data structures and algorithm performance, and survival strategies for batch programming environments. I worked full time from my sophomore year throughout college and graduate school in the university’s medical research laboratories to set up and maintain instrumentation, assist in research studies, and perform data reduction123. While I had many offers from other firms, Boeing had a huge attraction advantage given its headquarters in the beautiful outdoors of the Pacific Northwest.
My first Boeing assignments introduced me to the world of real-time data processing using minicomputers - now relics in museums - but at the time were signaling a major shift away from computing in centralized data centers onto dedicated devices for specialized applications. This shift also enabled broader applications to the specialty area of real-time data processing, an emphasis that would dominate my career.
Upon my arrival, Boeing was actively managing the concurrent development of two new aircraft, the 757 and 767. The cancellation of Boeing’s SST in the early 1970’s - which led to the infamous Boeing Bust - was finally behind the company, and it was again the dominant force in commercial aviation, with a formidable reputation in the industry for its innovation and engineering excellence. As the company has often been known to do, it bet its entire future on developing these two new aircraft to take maximum advantage of the emergence of digital avionics. By automating the monitoring and operations performed by the flight engineer, this innovation enabled reductions in crew which had enormous economic benefits to airlines, a bellwether for the effects of artificial intelligence in many areas today.
Neither of these new aircraft would be allowed to leave the ground even for test flights until the structural integrity of the aircraft designs had been adequately demonstrated and bought off by certification authorities. While similar tests had been performed on prior test programs - most recently for the 737 (which entered service 11 years earlier) and 747 (which entered service 9 years earlier), the retained knowledge from these efforts was limited as 60% of the company’s workforce had been laid off earlier in the decade. These ‘boom or bust’ cycles were a part of the Boeing culture and contributed to the strength of unionization of most manufacturing and engineering employees in the Puget Sound region.
One of my earliest assignments was to write software for a new array processor which the hardware test group had recently procured - at considerable expense - to support this structures testing. Interestingly, in this era, hardware was expensive, and software was comparatively affordable. The resulting integrated system needed to collect about one thousand samples of raw data in real time at ten millisecond data rates and convert those samples into engineering units, which were necessary to calculate stress and strain in support of full-scale aircraft fatigue testing, a 24x7 operation. These test systems required continual monitoring of these critical parameters to ensure values were within pre-established safety margins and to provide accurate data necessary for aircraft verification. Any failures in this process could severely compromise both programs, with few options for recovery. The endeavor had considerable skin in this game.
The group I was in was growing in support of these efforts; after two years, I was promoted to lead engineer of a team of six software engineers. Our team built out multiple data collection and analysis systems to support these testing efforts. In parallel, I was assigned to implement a multi-vendor communications network to exchange data across our many test facilities across Boeing’s Commercial Airplane division. We had few tools available in those days besides the compilers, editors, and operating systems that were provided by the computing manufacturers. Processes were a mechanism for supporting concurrency, rather than instructions on how to work. My team worked extremely well together, and we all experienced the joy that results from collaborative success as the team was able to overcome many challenges.
Despite the added workload from this team leadership, I retained my responsibilities as a technical contributor on the team, along with mentoring, overseeing, and integrating the efforts of other team members and interfacing with management. The array processor had been delivered and required crafting custom device drivers and interfaces for analysis, graphics, and reporting software which was being built by my team in cooperation with our customers. We wrote systems software in assembly language and applications in the programming language of choice in that day, Fortran.
My take on this new device was that no one really understood how this array processor would work (or even if it would be adequate to the task) prior to the purchase. Critical elements like the bandwidth of the buses involved, the write performance of the disks, and the time budget required for software functions to perform the necessary calculations had not been considered; our mindset was buy the best and caulk the rest. My personal plan was to explore, prototype, experiment, evaluate, learn, and iterate. Many late nights were involved, debugging code generation errors from immature compilers, troubleshooting problematic hardware, and ensuring my own code was adequate to the demands of the real-time environment.
During this time, I also introduced a new customer communications construct into our group which we ended up calling a Project Review Board. This PRB involved key customer subject matter experts in weekly meetings to discuss near-term plans and consider options in play at that time for the development team. Each month I also had a one-on-one ‘accountability review’ with the broader organization’s chief engineer to discuss the progress our team had made over the previous month. These habits of iterative development, team engagement, management coordination, and customer collaboration would form the essence of modern agile software development practices and serve me well for the rest of my career.
Later in my career, Boeing decided they no longer wanted to keep copies of personnel files in the company, and they sent mine to my home, which included a listing of the code which I wrote during this period, and which I’ve been able to review over 4 decades later (figures 1 and 2).
The critical code had to be written in assembly language (something few programmers use today). My underlying design objective was to hide secrets from customers and application programmers about where data was being kept in memory and in the array processor, with the goal of providing flexibility to enable those decisions to be ‘tunable’ to optimize performance for different situations. I also wanted to utilize constructs that would be familiar to end users. The design documentation was embedded in the code, which was my practice; a sample of this is provided in Figure 1 below.
Programming in assembly language can be challenging because that level of coding provides little structure for decision logic or program flow. As a team, we decided to adopt a convention of peer reviewing our designs once they were sketched out using a standardized pseudocode syntax. I worked with another engineer to build a preprocessor for this pseudocode which could check for errors and format these designs according to this structure and then migrate that pseudocode into comments in the assembly language code, continuing to prettyprint the text with indents and line wraps to promote readability according to the control structure of the pseudocode. I then built a series of assembly language macros which matched that same control structure and which autogenerated the assembly jumps necessary to implement that structure. Figure 3 below provides a sample code fragment from what that looked like:
The state of technology has advanced to the point today where such techniques are rarely visible to programmers; the capabilities of operating systems and tool chains are really the foundational miracles of our world. However, there are always space and time limitations somewhere, and good engineers find ways to battle through such constraints and produce solutions that stand the tests of time. And that is a most gratifying accomplishment.