Improving Program Success Through Systems Engineering Tools in Pre-Milestone B Acquisition Phase

To print a PDF copy of this article, click here.

Author: Daniel Deitz, Timothy J. Eveleigh, Thomas H. Holzer, and Shahryar Sarkani

Today, programs are required to do more with less. With 70 percent of a system’s life-cycle cost set at pre-Milestone B, the most significant cost savings potential is prior to Milestone B. Pre-Milestone B efforts are usually reduced to meet tight program schedules. This article proposes a new Systems Engineering Concept Tool and Method (SECTM) that uses genetic algorithms to quickly identify optimal solutions. Both are applied to unmanned undersea vehicle design to show process feasibility. The method increases the number of alternatives assessed, considers technology maturity risk, and incorporates systems engineering cost into the Analysis of Alternatives process. While not validated, the SECTM would enhance the likelihood of success for sufficiently resourced programs.

This article examines the importance of developing a robust Analysis of Alternatives (AoA) early in the concept phase of the acquisition program (prior to Milestone B) and the effects such development may have on program success. While current statutes require that program managers complete an AoA for all Acquisition Category (ACAT) programs, the quality of the AoA is the predominant indicator for program success and consists of more than just completing a study (Government Accountability Office [GAO], 2009a). In 2008, the Department of Defense (DoD) had 96 major defense acquisition programs, which experienced a cost growth of $296 billion and an average schedule delay of 22 months (GAO, 2009a). The GAO completed a study in 2009 where it identified one of the key causes for this cost and schedule growth as the mismatch between the requirements of the systems and the resources to provide them (GAO, 2009a). GAO further stated that programs enter the acquisition process with requirements that are not fully understood, cost and schedule estimates that are based on optimistic assumptions, and a lack of sufficient knowledge about technology, design, and manufacturing.

The DoD has a history of rushing programs into development or production that are not ready due to various program constraints. The Joint Strike Fighter was intended to produce an affordable aircraft, but ended up being the most expensive aircraft program in DoD with over $200 billion for 3,000 aircraft. GAO attributed a major factor for the cost overrun to the program’s premature entry into the engineering, manufacturing, and development phase prior to the maturation of critical technologies (GAO, 2001). The Navy has entered into shipbuilding contracts without fully maturing component technologies, resulting in a 193 percent cost growth on Littoral Combat Ship 1 and a 52-month delay on Landing Platform Dock 17 (GAO, 2009b). This rush is not just on large ACAT I programs, but also on smaller ACAT programs (Pincus, 2012). The Navy Organic Airborne and Surface Influence Sweep (OASIS) program just experienced a cost increase from $55 million to $135 million, with an 8-year delay in fielding. This system still has not met the requirement to continue operating after being hit by a shock wave from a mine or ordnance explosion (Pincus, 2012). The latest results from the last Department of Defense Inspector General (DoDIG) study indicated OASIS met only 65 percent of its shock requirement and would not work (DoDIG, 2012).

This article defines a new Systems Engineering Concept Tool and a five-step system engineering Method (SECTM) that we developed to increase the robustness of AoAs. We based the SECTM design on the finding from the GAO (2009a) study that examined 32 DoD programs, and the impacts that the quality of the AoA can have on program success. We applied SECTM to a UUV concept design to show the feasibility of implementing the process. SECTM includes a Systems Engineering Concept Tool based off genetic algorithms to quickly explore the design solution space. While SECTM cannot be validated until implemented by acquisition programs, it is expected to increase the likelihood of successful programs that, if sufficiently resourced, can deliver on time and on budget.

Importance of Early Information

Within the increasingly constrained fiscal environment in which the DoD must operate, program life-cycle cost control is especially important. All DoD programs, no matter which ACAT level is involved, follow a program path that has an impact on life-cycle costs. Smaller ACAT programs can streamline or skip minor steps, but the overall acquisition process is the same. The Defense Acquisition University has defined life-cycle cost across the various program milestones as shown in Figure 1 (Defense Systems Management College, 1990). Only 10 percent of the program’s life-cycle cost is invested during the system’s research and development phase up to the system’s initial operational capability; however, this may be the most important 10 percent of the system’s life-cycle cost. As this phase commits 70 percent of the program’s life-cycle costs, focusing significant time and effort to assure that all alternatives are considered is very important.

Figure 1. Defense Acquisition University Defined Life-cycle Across Various Program Milestones

Current Analysis of Alternatives

In today’s environment, program managers are encouraged to move as quickly as possible to meet urgent operational requirements, replacement schedules, or to save time. Because the majority of the pre-Milestone B work is level of effort, shortening this effort is easier than shortening the design and fabrication work. While this approach may be appealing to many program managers and requirements officers, the acquisition efforts leading to Milestone B set the foundation for the program. The work in this phase defines the acquisition strategy and life-cycle cost.

In 2009, GAO analyzed 32 major defense acquisition program starts since fiscal year 2003. That analysis is summarized in Table 1 (GAO,  2009a).

Table 1. Comparison of the Scope of Alternatives Considered with Program Cost and Schedule Growth

Number of programs with cost or schedule growth
Scope of alternatives Low Moderate High
No AoA conducted 7 0 3
AoA included broad scope of alternatives 7 1 1
AoA included narrow scope of alternatives 4 1 8

Source: GAO.
aCost growth: High = 25 percent or greater growth in development cost (or procurement costs for nondevelopmental programs) from initial baseline to current estimates, Moderate = 10–24 percent growth in development cost (or procurement costs for nondevelopmental programs) from initial baseline to current estimates, Low = less than 10 percent growth in development cost (or procurement costs for nondevelopmental programs) from initial baseline to current estimates.
Schedule growth: High = greater than 12-month delay for the initial operational
capability date or acquisition cycle, Moderate = 7- to 12-month delay for the initial
operational capability date or acquisition cycle, Low = less than 7-month delay for the
initial operational capability date or acquisition cycle.
b Narrow scope of alternatives = 2–5 alternative within one concept; broad scope of
alternatives = 8–26 alternatives within one concept, or alternatives within multiple concepts.

Of the 32 major DoD acquisition programs, 10 programs did not complete a formal AoA. For at least seven of those, this may have been appropriate since they were modernization or evolutionary programs. The Defense Acquisition Guide, which states that an AoA should focus on the end-state solution, contains recommendations on a single development or evolutionary development path (Defense Acquisition University, 2012). The Milestone Decision Authority can waive the requirement for a new AoA for incremental or modernization efforts included in previous analyses. The Navy Standard Missile SM6 is an example of an evolutionary acquisition program where block increments were used to incrementally reach the final capability, thereby negating the necessity for an AoA. Thirteen major acquisition programs conducted a narrow scope AoA where over 60 percent of the programs experienced significant cost or schedule growth. Nine major acquisition programs conducted a broad scope AoA where only one of these programs experienced a significant cost or schedule growth. This GAO (2009a) study showed that broader scope AoAs had less cost and schedule overruns.

Because the majority of the pre-Milestone B work is level of effort, shortening this effort is easier than shortening the design and fabrication work.

Let us highlight one program’s AoA process. The Air Force needed to replace its KC-135 tanker. This was a high-visibility major ACAT I defense program for the Air Force. The KC-135 provided 80 percent of U.S. air refueling capability that enabled airpower to be deployed and sustained overseas in a timely manner. The fleet of KC-135s was reaching 50 years of age and becoming increasingly costly to maintain and operate. The replacement program for these aircraft was expected to be close to $200 billion (RAND, 2006), and 6 months were allocated for the AoA.

The KC-135 AoA was required to study the amount of fuel the aircraft could supply along with the times and locations in a set of mission scenarios (RAND, 2006). The AoA met these criteria through analyzing four major aircraft classes and seven different methods to procure those classes. However, the AoA was focused on only one major objective: life-cycle cost. The AoA assumed that all threshold requirements must be met, so no analysis was conducted to see if any single requirement was driving the cost. In addition, the AoA did not look at the technology risk of the program to predict the level of uncertainty that can drive program overruns late in the design.

The GAO (2009a) report found that narrowing the AoA scope to life-cycle cost did not enable the identification of the most promising alternative, and reducing the AoA schedule did not allow enough time to complete a thorough analysis. The GAO study recommended that DoD develop guidance for conducting robust AoAs to adequately select an alternative (GAO, 2009a).

Systems Engineering in the Pre-Milestone B Acquisition Phase

Program managers and resource sponsors are under increasing pressure to perform at a higher level with less resources. It appears unlikely that increasing either the timeline or the cost of conducting an AoA is an option. We propose to use our SECTM in the AoA process to thoroughly evaluate additional alternatives in the same AoA timeline.

The systems that DoD acquires have become more complicated, and quantifying the effect that each requirement has on these systems is becoming increasingly difficult. As the DoD strives to adopt more commercial practices, it will need to adjust its acquisition processes. Unlike the DoD, the commercial industry focuses on the market and the price point to enter into that market. Most commercial industry program/project managers attempt to find the best value for the customer by providing the most capability for a set price. In today’s shrinking defense budget, a more commercial strategy may be needed to keep the same force levels and capability despite reduced funding. According to Navy Admiral Jonathan Greenert (Chief of Naval Operations, 2012):

We can no longer afford, strategically or fiscally, to let the perfect be the enemy of the good—or the good enough—when it comes to critical war fighting capability. (p. 7)

Systems engineering provides the rigor needed to handle the increasing complexity of today’s DoD systems. We are moving from lowest cost for a set threshold performance to simultaneously minimizing or maximizing multiple objectives like minimizing cost, maximizing performance, and minimizing program risk. In multiple-objective analysis, multiple solutions exist that are each optimal, since they are at least as good as any other solution for some weighted combination of the multiple objectives. For that reason, these solutions are referred to as nondominated, as they each have no other solution that dominates for at least one weighted combination of the objectives. The set of all nondominated solutions is referred to as the Pareto front. Figure 2 is an example of a Pareto or nondominated solution where the design solutions are shown in orange and the optimal solutions form a line shown in green (Brown, 2003). In a Pareto optimal designed system, the design can trade off cost versus risk to find an optimal solution. As parameters are varied in one optimal solution, they create other optimal solutions if the solution improves in meeting at least one objective. Therefore, for a solution to be optimum, it can only decrease cost to the point where it has a negative effect on performance or program risk.

Figure 2. Systems Engineering Concept Tool and Method (SECTM)

Using multiple-objective analysis, we developed a new tool and designed a method (SECTM) to address GAO’s recommendation that DoD AoAs need to investigate a broader scope of alternatives to increase their robustness (GAO, 2009a). The SECTM increases breadth of the alternatives considered, investigates program risk based on technology selections, and addresses systems engineering complexity in the cost estimate, as shown in Figure 3. The proposed approach assesses the technologies, defines the system metrics, provides a tool to evaluate the alternatives, and can provide the stakeholders with an assessment of optimal alternatives. The alternatives that appear to be within the available resources could proceed to the formal DoD AoA process.

The steps in our proposed process are detailed in the following discussion.

Figure 3. Example of a Pareto or Nondominated Solution

Step One—Assess Availability of Current Technology

The first step in the proposed approach is to assess the current technology available and develop models of those key technologies or subsystems. This will allow a wide net to be cast for investigating technologies, typically using Technology Readiness Levels (TRL) 1–9. In 2001, the Deputy Under Secretary of Defense for Science and Technology endorsed the use of TRL in new major acquisition programs. DoD Instruction (DoDI) 5000.02 describes TRLs from a systems perspective and states that they are to be used for both hardware and software (DoD, 2008). While to date, they have been used in the criteria for gate reviews, they do not fold in program risk. Subsystem concept models should be created to represent system performance, cost, and risk, and include TRL evaluation.

The cost of doing systems engineering is becoming a significant cost factor due to the increased complexity of today’s systems.

Step Two—Define Objectives for Alternatives

The second step, which can occur in parallel to step 1, is the definition of high-level objectives (and associated metrics) for the alternatives. For robustness, there should be at least three primary objectives considered: technical performance, cost, and risk (GAO, 2001). These objectives will be used to rank the different alternatives and provide recommendations on the set of optimal solutions in the next step.

Technical performance and cost objectives are part of the standard AoA process and should continue to be defined. In addition to cost and performance, we recommend using Technology Maturity Risk as a new objective. The GAO (2009a) report states that inadequate technology maturity is a key factor in program cost and schedule overruns. The time has come to explicitly consider Technology Maturity Risk in the AoA to increase program success.

Researchers at the University of Southern California Center for Software Engineering have proposed an approach that includes technology maturity risk. They report that TRL maturity has both positive and negative aspects. Higher, more mature technologies can have a greater risk of obsolescence or the possibility of a leap-ahead technology during the life of the DoD system. Lower, less mature technologies have greater development cost and schedule risk. These researchers have proposed a new Technology Maturity Risk function based on the TRL of a technology, the maturity of the technology in a system, and the risk of obsolescence. While Technology Maturity Risk has been considered in the past, Valerdi developed a new model that links the Technology Maturity Risk to a programmatic cost (Valerdi, Boehm, & Reifer, 2003). From Valerdi’s study, which included efforts of over 40 systems engineering experts, this Technology Maturity Risk has also been associated with a program cost impact. Table 2 shows the rating scale for Technology Maturity Risk (Valerdi & Kohl, 2004).

Table 2. Technology Maturity Risk

Viewpoint Very Low Low Nominal High Very High
Lack of Maturity Technology proven and widely used throughout industry Proven through actual use and ready of widespread adoption Proven on pilot projects and ready to roll-out for production jobs Ready for pilot use Still in the laboratory
Lack of Readiness Mission proven (TRL 9) Concept qualified Concept has been demonstrated (TRL 7) Proof of concept validated (TRL 5 & 6) Concept defined (TRL 3)
Obsolescence (Obsolescence not an an issue) (Obsolescence not an an issue) Technology is the state-of-the-practive; emerging technology could compete in future Technology is stale; new and better technology is on the horizon in the near-term Technology is outdated and use should be avoided in new systems; spare parts supply is scarce
Cost Multiplier 0.68 0.82 1.0 1.32 1.75

Many programs underestimate the cost of the large systems engineering effort required to develop complex systems. Therefore, in addition to the life-cycle cost models of the individual systems (aircraft, ship, vehicle, weapons, information technology, etc.), the cost models need to consider the systems engineering cost. While advocating no particular cost modeling tool, the authors surmise that, to properly determine life-cycle costs, systems engineering costs must be considered in the life-cycle cost calculation. In 2003, Valerdi developed the Constructive Systems Engineering Cost Model (COSYSMO) for the purpose of estimating the systems engineering effort needed for large complex systems (Valerdi, Boehm, & Reifer, 2003). His analysis is based on four categories: product, platform, personnel, and project (with technology risk being a driver). With assistance from the International Council on Systems Engineering, the COSYSMO model has been validated with industrial partners while new lessons learned are continually incorporated (Valerdi, Rieff, Roddler, & Wheaton, 2007). The cost of doing systems engineering is becoming a significant cost factor due to the increased complexity of today’s systems.

Step Three—Apply a Systems Engineering Concept Tool

The heart of SECTM is our Systems Engineering Concept Tool. The subsystems models from step 1 and the objectives from step 2 feed our Systems Engineering Concept Tool. We recommend that a genetic algorithm solver (see further discussion) be used because the user can select the number of alternatives to be considered, and genetic algorithms provide a good estimate of the optimal solution set. For complex systems with 10 or more critical design parameters, the number of different solutions can range from 10 to 100 billion, which is far too many to investigate. A genetic algorithm solver quickly defines the solution space and provides a near-optimal solution set in a small number of iterations (Deb, 2001). A classical optimization problem would compare each possible solution pairwise, for which there may be 10 billion comparisons. Genetic algorithms can provide a good estimation of the optimal solutions with a population size as small as 100 and converge as quickly as 10 generations from the results of the UUV example described later. Deb (2001), a recognized expert in genetic algorithms, states that genetic algorithms have tremendous advantage over classical search techniques because genetic algorithms move the entire optimal population toward the optimal solutions instead of a single solution.

Genetic algorithms were developed to imitate the processes that evolve in living beings, and
the algorithms allow the designs to evolve each generation to better meet the identified objectives.

Genetic algorithms were developed to imitate the processes that evolve in living beings, and the algorithms allow the designs to evolve each generation to better meet the identified objectives. Even if they do not use this formal methodology, designs typically evolve just with trial and error (Eddy & Lewis, 2001). The heart of genetic algorithm research began with Schaffer in the 1980s. Now, many genetic algorithms are available that can be applied to this problem due to prior research (Schaffer & Grefenstette,1985; Zitzler, Deb, & Thiele, 2000; Horn, 1997). A few common algorithms include the Vector Evaluated Genetic Algorithm (VEGA), Nondominated Sorting Genetic Algorithm (NSGA II), and Niched Pareto Genetic Algorithm II (NPGA) (Zitzler, Deb, & Thiele, 2000). VEGA, one of the first Pareto genetic algorithms from the 1980s, works by assigning a randomly selected single objective to each member of the population. NPGA was developed by Horn and Nafpliotis in 1993, and improved on the selection process determining the dominance of randomly selected groups of individuals in the population (Coello Coello, 2000). NSGA II was developed from work by Srinivas and Deb (1994) in 2000, and improved upon the basic genetic algorithm by sorting the population in multiple-level solutions, starting with the nondominated and binning them into levels of domination until all solutions are binned. The level of domination is used to modify the fitness of individuals. This allows for quick and computationally efficient algorithms compared to other methods (Coello Coello, 2000), which is why we selected NSGA II for this research. Genetic algorithms have been used to solve many complex problems, especially in aircraft and shipbuilding where many objectives compete with each other. Figure 2 shows an example of Pareto front trading off effectiveness or performance versus cost. The feasible region is the large number of possible alternatives in orange. The nondominated solutions are the set of optimal solutions. For multiple-objective problems, two solutions are possible: one dominates (or is better than the other) or nondominated (each solution is equally good as one another). Defined by Goldberg (1989), a solution is considered nondominated when an objective cannot be increased without reducing the other objectives. In complex systems, rarely is there one optimal solution, but rather a set of optimal solutions. The stakeholders need to make the final choices between optimal solutions that best meet their needs. The goal of our Systems Engineering Concept Tool is to identify those optimal solutions.

The Systems Engineering Concept Tool should, at a minimum, include three primary objectives: technical performance, cost, and risk. Many AoAs today only consider the cost to meet the threshold requirement, whereas SECTM would allow key performance parameters to be separate objectives, and cost and risk could be traded among those key parameters.

Step Four—Presentation of Optimal Alternatives to Stakeholders

The fourth step in the proposed process shown in Figure 3 is to present the optimal alternatives developed from the Systems Engineering Concept Tool to the stakeholders. Since one optimal solution is rarely applicable in complex systems, an Executive Steering Group (ESG) should narrow the set of optimal solutions to those that fall within the resources available and the program constraints. Through the use of tradeoffs, SECTM will provide a set of optimal solutions that meet the metrics defined in step 2. Since these solutions are equal mathematically, the ESG needs to identify or narrow the “best solutions” dependent on preferences and experience (Faulkenberg & Wiecek, 2010). These narrowed solutions should then undergo a detailed analysis by subject matter experts. The ESG can also decide to change the metrics to refine this analysis if none of the alternatives are appropriate.

Step Five—Detailed Analysis (Similar to DoD’s AoA Process)

The last step is to take the narrowed set of optimal alternatives and complete a detailed analysis of each alternative. This step is similar to the DoD AoA process, which uses a set of subject matter experts and increased fidelity models and simulations to determine and subsequently recommend the best alternatives. The analysis in step 5 will use high-fidelity physics models that are significantly more detailed than the subsystem concept models used in step 1.

Summary of the Five Steps

The developed methodology described in the preceding 5 steps is anticipated to increase the robustness of the DoD pre-Milestone B Phase AoA by:

  • Widening the solution space investigated within the time and personnel constraints;
  • Incorporating the Technology Maturity Risk; and
  • Incorporating the cost to mitigate Technology Maturity Risk and the level of systems engineering needed for complex DoD programs.

Application of SECTM to Unmanned Systems Concepts

In this section we demonstrate the use of SECTM on unmanned systems that have a strong appeal in the DoD environment. Unmanned systems take the DoD’s most valuable asset—its personnel—and remove them from dull, dirty, and dangerous tasks. These systems demonstrably reduce the forward deployments of our military personnel, thereby increasing the quality of life for our soldiers and their families.

Unmanned aerial vehicles (UAV) have been used for many years with high success in the war against terrorism. Secretary of the Navy Ray Mabus stated his priority in maintaining the competitive edge by moving beyond pilotless UAVs to fielding unmanned undersea vehicles (UUV), as well as surface vehicles (Mabus, 2010). UUVs will provide a new capability without significant experience or analysis to bound AoA scope. Since UUVs may be the next big acquisition of unmanned systems, they are a good test case for SECTM. The following discussion reapplies the steps defined earlier using an analysis of UUV designs as an example.

Step One—Assess Availability of Current Technology

The first step in applying SECTM was to analyze the UUV subsystems and to determine the critical technologies in each subsystem. Subsystem models were created from core hydrodynamic texts and UUV literature from Massachusetts Institute of Technology and Southampton Universities (Furlong, McPhail, & Stevenson, 2007). We developed a basic UUV cost model using the Naval Sea Systems Command cost estimation handbook, and adapting a submarine cost model and systems engineering cost models (Valerdi, Boehm, & Reifer, 2003).

We identified critical technologies for achieving endurance that were based on experience with the Autosub UUV (Furlong, McPhail, & Stevenson, 2007) being designed for an endurance of 5,000 meters and buoyancy-driven UUVs. For simplicity of this example, the two driving technologies are the energy density of the primary power system and the hotel load (for example heating, computing, power distribution). Most current UUV systems use batteries, which are a high TRL (mature), but low-energy density. However, high-energy density power systems like fuel cells and combustors are being developed and show promise at low TRLs. We completed a market survey to look at the different battery technologies and their energy density as a function of TRL. Hotel power was linked to one primary technology—computer processors. We used current quad-core processors as the standard processor at TRL 8. New gaming and cell core processors are being developed that have the potential to reduce the processing power by a factor of four, but these are only at TRL 3. Once again, we completed a market survey and created a model to link processing power with a TRL.

Step Two—Define Objectives for Alternatives

Step Two defines the objectives for the system Pareto analysis. We used the 2004 Navy UUV Master Plan as a guiding document to determine the UUV design objectives (Department of the Navy, 2004). The first objective was to maximize the endurance or range of a UUV to be able to perform Navy missions like mine warfare and intelligence, surveillance, and reconnaissance. The second design objective was to minimize the UUV’s volume. This is important for integration of the UUV onto existing Navy platforms since larger UUVs may not fit on many Navy ships. The third and fourth objectives were cost and technology risk.
Figure 4 illustrates the 10 different parameters we considered in this UUV analysis and links those parameters to each objective.

Figure 4. SECTM Objectives and System Parameters

Early communication with stakeholders on potential alternatives can facilitate a better
understanding of the requirements.

Step Three—Apply a Systems Engineering Concept Tool

We chose the NSGA-II genetic algorithm developed by Deb (Deb, Pratap, Agarwal, & Meryarivan, 2002) for our basic genetic algorithm solver because of this algorithm’s computational efficiency. NSGA-II’s computational efficiency can be approximated by the formula: f(M*N2) as opposed to other sorting algorithms, which use f(M*N3) where M is the number of objectives and N is the genetic population size. For the UUV example where M = 3 and N = 100, NSGA-II saved 2,970,000 computations. We programmed equations for each of the design objectives into Matrix Laboratory, or MATLAB programming language using the NSGA-II algorithm for the optimization.

Since the population size is variable, it can be selected by the users. Increased population size provides more points on the Pareto front to better identify design trends. However, increased population size will square the number of computations required.
For this example, there are 10 basic design parameters (r). For simplification, if each design parameter had only 10 different applicable values (n), then through pairwise comparison:

Estimated number of comparison = nr

For this example, n = 10 and r = 10, which is 10 billion combinations that would have to be analyzed. This is far too many to accomplish in just a few months; however, the use of a genetic algorithm solver reduces the number of processes significantly. The genetic solver starts with random solutions. Those solutions that have a higher match to the objectives are selected for regeneration and combined together to create a new generation. This process is continued and mimics the way living species survive and adapt to the environment. Genetic algorithms usually can converge in 10 generations; therefore, the amount of calculation needed is the population size times the number of generations:

Number of designs: population x generations = 100 x 10 = 1,000

The results of this application show that for the UUV design discussed here, which is fairly simple compared to many DoD systems, the solution space of 10 billion different design combinations can be approximated by a population size of 10 or 100 with less than 1,000 design iterations using the SECTM. SECTM is a very efficient way to determine a set of optimal alternatives to present to the stakeholders.

Early communication with stakeholders on potential alternatives can facilitate a better understanding of the requirements. Today’s systems are so complex and highly integrated, that it is impossible to understand the large impacts that small changes can make without the use of analysis tools. SECTM provides a visualization of the tradeoffs of risk, cost, and technical performance. These tradeoffs are very important to the success of the program. Using current practices, stakeholders are not presented with enough data to make good decisions. Steps 4 and 5 were not completed in this example as they feed the DoD AoA process and were not needed to show the feasibility of the SECTM.


In today’s reduced budget and constrained fiscal environment, making acquisition decisions that provide the best value to the nation’s armed forces and the DoD is extremely important. Over 70 percent of a system’s life-cycle cost is determined by Milestone B; therefore, the largest impact can be made during these early program stages. Unfortunately, this is where a large majority of programs streamline, reduce, or cut activities to save time and funding. Out of 32 programs reviewed by the GAO (2009a), 60 percent of the programs that completed limited scope in their AoAs experienced significant cost and/or schedule overruns compared to less than 10 percent in those programs that completed a robust AoA.

Applying the SECTM in the pre-Milestone B Acquisition Phase is an option to increase the AoA’s robustness without significantly increasing cost or time. Current processes use a team of experts to analyze a few predetermined alternatives (three to 10 for a typical acquisition program) and primarily conduct interviews to make subjective analyses. This article proposed a SECTM to be used in the AoA process to help determine or down-select the few alternatives that are investigated in depth by an AoA team. When we applied our SECTM to a UUV, we were able to reduce 10 billion design combinations to a set of only a few optimal solutions. This initial systems engineering step can be done rapidly using modeling and simulation tools, and by using the engineering process to down-select the alternatives instead of a steering group committee process.

This article also presented the importance of the pre-Milestone B Acquisition Phase in setting the foundation for the success of the program. This methodology presented a way to increase the robustness of the alternatives considered in pre-Milestone B acquisition documentation (primarily AoAs) and incorporates the following new aspects:

  • Widening the solution space investigated within the time and personnel constraints;
  • Incorporating the Technology Maturity Risk; and
  • Incorporating the cost to mitigate Technology Maturity Risk and the level of systems engineering for complex DoD programs.

While pre-Milestone B efforts only account for less than 10 percent of the total life-cycle cost, they are the most important 10 percent of funding because they set the acquisition program on a sound foundation and business case. Errors in this phase cost between three and 10 times more to fix in later phases. The GAO recommended to the DoD that new criteria should be set for execution of AoAs, with the DoD agreeing to the recommendation. The approach proposed in this article is a way to increase the robustness of DoD’s AoAs.

To print a PDF copy of this article, click here.

Author Biographies

Dr Daniel DeitzDr. Daniel Deitz is an unmanned undersea systems subject matter expert at the Office of Naval Research with 13 years of Department of Navy acquisition professional experience. He holds an MS in Systems Engineering from Johns Hopkins University, a BS in Mechanical Engineering from Pennsylvania State University, and a PhD from The George Washington University. Dr. Deitz holds several Defense Acquisition Workforce Improvement Act certifications.

(E-mail address:

Dr Timothy J EveleighDr. Timothy J. Eveleigh is an adjunct professor of Engineering Management and Systems Engineering at The George Washington University and an International Council on Systems Engineering-certified Systems Engineering Professional. He has over 30 years’ industry experience working DoD/intelligence community information technology acquisition challenges, research and development, and enterprise architecting. Dr. Eveleigh has a 30-year parallel career as an Air Force Reserve intelligence officer and developmental engineer, focused on command and control integration.

(E-mail address:

Dr Thomas H HolzerDr. Thomas H. Holzer is an adjunct professor of Engineering Management and Systems Engineering at The George Washington University. He was the director, Engineering Management Office, National Geospatial-Intelligence Agency, with over 35 years’ experience in systems engineering and leading large-scale information technology programs. Dr. Holzer holds a Doctor of Engineering Management and Master of Science in Engineering Management from The George Washington University; and a Bachelor of Science in Mechanical Engineering from University of Cincinnati.

(E-mail address:

Dr Shahryar SarkaniDr. Shahryar Sarkani is an adjunct professor in the Department of Engineering Management and Systems Engineering at The George Washington University, with over 20 years of experience in software engineering focusing on architecture and design. Dr. Sarkani holds a Doctor of Science in Systems Engineering from The George Washington University, a Master of Science in Mathematics from University of New Orleans, and a Bachelor of Science in Electrical Engineering from Louisiana State University.

(E-mail address:


Brown, A., & Salcedo, J. (2003). Multiple-objective optimization in naval ship design. Naval Engineers Journal, 115(4), 49–61.

Chief of Naval Operations. (2012, March). FY 2013 Department of Navy posture. Statement of Admiral Jonathan Greenert, Chief of Naval Operations, Before the 112th Congress. Retrieved from

Coello Coello, C. (2000). An updated survey of GA-Based Multiobjective Optimization Techniques. ACM Computing Surveys, 32(2), 109–143.

Deb, K. (2001). Multi-objective optimization using evolutionary algorithms. Chinster, England: Wiley.

Deb, K., Pratap, A., Agarwal, S., & Meryarivan, T. (2002). A fast and elitist multiobjective genetic algorithm: NSGA II. IEEE Transactions on Evolutionary Computing, 6(2), 182–197.

Defense Acquisition University. (2012). Defense Acquisition Guidebook. Retrieved from

Defense Systems Management College. (1990). DSMC systems engineering management guide. Retrieved from

Department of Defense. (2008). Operation of the defense acquisition system (DoDI 5000.02). Retrieved from

Department of Defense Inspector General. (2012). Acquisition of the Navy organic airborne and surface influence sweep need improvement (Report No. DoDIG-2012-101). Retrieved from

Department of the Navy. (2004). The Navy unmanned undersea vehicle (UUV) master plan. Retrieved from

Eddy, J., & Lewis, K. (2001). Effective generation of Pareto sets using genetic programming. Proceedings of DETC’01, ASME 2001 Design Engineering Technical Conference and Computers and Information in Engineering Conference, Pittsburgh, PA, September 9–12, 2001.

Faulkenberg, S., & Wiecek, M. (2010). Quality discrete representation in multiple objective programming. Optimization and Engineering, 11(3), 423–440.

Furlong, M., McPhail, S., & Stevenson, P. (2007). Concept for an ultra-long-range survey class AUV. Proceedings of IEEE Oceans 2007-Europe Conference, Hampton, UK, June 18, 2007.

Goldberg, D. E. (1989). Genetic algorithms in search, optimization, and machine learning. Reading, MA: Addison-Wesley Professional.

Government Accountability Office. (2001). Joint Strike Fighter acquisition: Mature critical technologies needed to reduce risks (Report No. GAO-02-039). Retrieved from

Government Accountability Office. (2009b). Best practices: High levels of knowledge at key points differentiate commercial shipbuilding from Navy shipbuilding (Report No. GAO-09-322). Retrieved from

Government Accountability Office. (2009a). Many analyses of alternatives have not provided a robust assessment of weapon system options (Report No. GAO-09-665). Retrieved from

Horn, J. (1997). The nature of niching: Genetic algorithms and the evolution of optimal, cooperative populations (Doctoral dissertation, University of Illinois at Urbana-Champaign). Retrieved from

Mabus, R. (2010, May). The SECNAV: Implementing policy, taking care of people. Seapower magazine interview by Managing Editor Richard R. Burgess and Editor in Chief Amy L. Wittman. Retrieved from

Pincus, W. (2012, June 18). Smaller defense programs need a closer look, too. The Washington Post. Retrieved from

RAND. (2006). Analysis of alternatives (AoA) for KC-135 recapitalization. Retrieved from

Schaffer, J., & Grefenstette, J. (1985). Multi-Objective learning via genetic algorithms. In A. K. Joshi (Ed.), Proceedings of the Ninth International Joint Conference on Artificial Intelligence (Vol. 1, pp. 593–595), Los Angeles, CA, August 18–23, 1985.

Srinivas, N., & Deb, K. (1994) Multiobjective optimization using nondominated sorting in genetic algorithms. Journal of Evolutionary Computing, 2(3), 221–248.

Valerdi, R., Boehm, B., & Reifer, D. (2003). COSYSMO: A constructive systems engineering cost model coming of age. Paper presented at the INCOSE 13th Annual International Symposium on Engineering Tomorrow’s World Today, Crystal City, VA, June 29–July 3, 2003.

Valerdi, R., & Kohl, R. (2004). An approach to technology risk management. Paper presented at the Engineering Systems Division Symposium, Massachusetts Institute of Technology, Cambridge, MA, March 29–31, 2004.

Valerdi, R., Rieff, J., Roddler, G., & Wheaton, M. (2007). Lessons learned from industrial validation of COSYSMO. Paper presented at the 17th Annual INCOSE International Symposium on Systems Engineering: Key to Intelligent Enterprises, San Diego, CA, June 24–28, 2007.

Zitzler, E., Deb, K., & Thiele, L. (2000). Comparison of multi-objective evolutionary algorithms: Empirical results. Evolutionary Computing, 8(2), 173–195.



Leave a Reply

Your email address will not be published. Required fields are marked *