By Bruce Harmon
We define near-substitutable systems as systems that have overlapping capabilities, but are substantially different in some dimensions. Competition between such systems is examined in the context of the dominant “Weapon System Franchise” model of competition for major defense acquisition programs (MDAPs).
Competition between near-substitutes can occur throughout the acquisition cycle. It is most commonly seen in pre-Engineering and Manufacturing Development Analyses of Alternatives (AoAs) where the cost-effectiveness of a variety of concepts as well as legacy systems and potential derivatives are compared. Our primary interest is in competition later in the acquisition lifecycle when the near-substitutes may already exist in some form. We explored the potential for competition between near-substitutable systems through three case studies:
C-17 and 747-400F/Non Developmental Airlift Aircraft (NDAA), Joint Air-to-Surface Standoff Missile and Standoff Land Attack Missile-Expanded Response, and KC-X: KC-767 and KC-45.
We found that near-substitutes can provide competitive pressure on incumbent systems. The KC X was exceptional in that it was a formal direct competition between near-substitutes leading to the award of a franchise. AoAs or other cost-effectiveness analyses can be pivotal in bringing attention to near-substitute systems. However, AoAs comparing alternatives to systems already in production tend to be ad hoc in nature. One way to further encourage competition between near-substitutes would be through a “rolling cost-effective analysis” process where the original AoAs would be updated as material changes become evident in an ongoing program or potential near-substitutes surface.
Competition between near-substitutes can occur throughout the acquisition cycle. We explored the potential for this type of competition through three case studies.
This paper examines competition among near-substitutable systems. Near-substitutes present an interesting case for competition; their competitive threat is often manifested early in the concept development stage of the acquisition lifecycle, but can also put competitive pressure on an existing weapon system franchise. Our approach was to use three case studies to illuminate the issues associated with competition between near-substitutable systems. The objective was to draw lessons to inform government actions and policies that could encourage such competition in the future.
Competition for a Weapon System Franchise
We considered competition among near-substitutes within the context of the “Competition for a Weapon System Franchise” case, as presented in Dominy et al. (2011). A franchise begins with development of the system, followed by serial production over a period that can continue for as long as 20 years. Typically, two (or very occasionally more) firms compete for an Engineering and Manufacturing Development (EMD) contract. The EMD process results in a detailed design of the system; design and production of the tooling and equipment—and sometimes facilities—needed to produce the system; and building of “production representative” units of the system for testing. Successive annual lots are then purchased using a series of separately negotiated contracts. Multi-year contracts (covering production of three to five annual lots) may be used in place of annual contracts once the system has reached maturity. These contracts ordinarily are typically placed with the firm that won the EMD contract, on a sole-source basis. This is the most common case; it is typical, for example, of major defense acquisition programs for aircraft, ships, tactical missiles, and combat vehicles. Competition among near-substitutes is a variant on this common case.
Definition of Near-Substitutable Systems
The definition of near-substitutable systems is the following: systems that have overlapping capabilities, but are substantially different in some dimensions. It is easiest to start with an example of what they are not. To achieve a military objective, it is sometimes possible to choose between systems that have no meaningful overlapping capabilities. An example would be the use of tactical jamming devices to penetrate enemy airspace, instead of designing stealth features into the aircraft platform. Yet a third way to achieve the same ends might be the use of long-range standoff weapons on conventional aircraft platforms. This mode of competition between non-overlapping alternatives clearly falls outside of our definition.
Thus we will limit the case of near-substitutes to be competitions among items of the same commodity class; e.g., aircraft versus aircraft, missile vs. missile. What then separates near-substitutes from perfect substitutes, i.e., the dual-sourcing variant of the weapons system franchise case? In bounding the definition in this direction we limit the near-substitute case to instances where systems were not originally designed to fulfill the same military requirement. Given this, a near-substitute may often be a non-developmental item that can be modified or repurposed to fill a military need.
This leads naturally to the question of where competition between near-substitutes fits within the overall acquisition process. We would expect this competition to begin in the early portions of the acquisition cycle, and continue at least though the performance of the initial Analysis of Alternatives (AoA). However, it would be rare (although not unprecedented) for competition between near-substitutes to occur as a result of a request for proposals (RFP) leading to direct competition for a franchise or as an alternative to a current franchise holder. Near-substitutes, however, can provide competitive pressure on incumbent or quasi-incumbent (heir-apparent) systems throughout the acquisition cycle.1 Decision makers can set up the competition between near-substitutes such that the incumbent is threatened with the loss of its franchise. AoAs or other cost-effectiveness analyses can prove pivotal in bringing attention to near-substitute systems; the role of cost-effectiveness analysis is an important topic in the case studies.
Literature on Competition among Near-Substitutes
Although we have found no analytic discussion specific to the near-substitute competition variant, it is useful to relate its unique attributes to the broader literature. In Rogerson’s (1994) survey article on the economics of defense procurement, he interprets the subject as a government regulatory problem with several distinguishing characteristics:
- The importance of research and development,
- Economies of scale in production, and
- The role of government as a sole purchaser.
Although all of these characteristics have some relevance to competition, we will concentrate on the first and third points. Due to the high costs of carrying multiple firms through EMD and production, competition is usually only feasible through the conceptual design and prototyping phases—the down select at this point establishes a franchise for a single winner. In our framework for competition between near-substitutes, the limitations imposed by high development costs and loss of economies of scale in production are less relevant: a near-substitute may already exist in some form and have an ongoing production base. This is an enormous advantage for the near-substitute as compared with an item yet to be developed.
We chose three case studies to help explore issues associated with competition between near-substitutes.
C-17 versus Commercial Cargo Aircraft
In the late 1980s and early 1990s, the C-17 program encountered substantial difficulties in terms of performance shortfalls, cost overruns, and schedule delays. Because of these difficulties, Congress directed Department of Defense (DoD) as part of its FY 1993–1994 Defense Authorization Act to conduct a Defense Acquisition Board (DAB) review of the program; included were an examination of C-17 requirements and affordability as well as the results from a new cost-effectiveness analysis (then referred to as a Cost and Operational Effectiveness Analysis, or COEA). At the time, the Air Force had planned to buy 120 C-17s, with 20 already funded (U.S. General Accounting Office, 1994).
The COEA was performed by the Institute for Defense Analyses (IDA); alternatives to the full 120 C-17 program included the procurement of Boeing 747-400F freighters (Boeing had yet to acquire McDonnell Douglas, the C-17 prime contractor at the time). Important characteristics of the C-17 and 747-400F are included in Table 1.
a Maximum number of aircraft on ground simultaneously in theater for the Major Regional Contingency-East scenario
The 747-400F has advantages in payload/range performance while the C-17 has substantially better Maximum on Ground (MOG) metrics, can deliver outsized cargo (primarily large armored vehicles) and performs military-specific missions such as air-drop and combat delivery. The 747-400F clearly fits into our definition of a near-substitute relative to the C-17 (the incumbent in this case).
The COEA posited alternative fleets with the same MTM/D as the planned fleet with 120 C-17s. While MTM/D is a static measure, the effectiveness modeling took into account dynamic effects, including the impact of limited airfield space in theater. The overall results were that a mixed fleet of C-17s and 747-400Fs (along with other aircraft in the planned force) had approximately equal cost-effectiveness when compared with the case with 120 C-17s. The COEA found significant synergies between the 747-400Fs and aircraft carrying outsize cargo (C17s and C-5s); the additional oversize and bulk carrying capacity of the 747-400Fs freed up space in the aircraft capable of carrying outsized cargo.
The December 1993 DAB review resulted in several actions. An RFP for a non-development airlift aircraft (NDAA) was released; the NDAA could be a new commercial freighter or refurbished/modified used aircraft. Boeing was the only respondent, with its C-33 (the military designation for the 747-400F). Parallel actions for the C-17 included directing management and manufacturing process improvements, as well as the approval of procurement through the fortieth aircraft. The C-17 was put on probation for two years, with the fate of the remaining 80 aircraft to be determined at the end of 1995. The C-33 provided DoD with a clear alternative if C-17 program improvements were not forthcoming, while placing competitive pressure on McDonnell Douglas.
By the end of 1995, the C-17 program had satisfied the DAB with its progress and an 80-aircraft multi-year procurement (MYP) was approved. The NDAA program was shelved. The general view was that the two-year probationary period was a success (Bolkcom, 2007). Comparisons of actual C-17 procurement cost with those presented in the 1993 COEA show the actual cost for the final 100 of the originally planned 120 aircraft substantially below that estimated by IDA, and slightly below the lower Air Force estimate ($21.8 billion versus $26.3 billion and $22.9 billion, respectively, in FY 1993 dollars for aircraft 21-120).
Joint Air-to-Surface Standoff Missile and Standoff Land Attack Missile-Expanded Response
In 1994, the Tri-Service Standoff Attack Missile (TSSAM) was cancelled because of cost overruns. However, the services’ requirement for an air-launched standoff precision weapon to attack well-defended high-value targets did not go away. The Joint Air-to-Surface Standoff Missile (JASSM) program was initiated (Milestone 0) in September 1995 as a joint Navy/Air Force program; the approach was to take the lessons learned from the TSSAM program and apply acquisition reform initiatives in order to produce an affordable missile with capabilities similar to those of the TSSAM.
The Navy’s Standoff Land Attack Missile-Expanded Response (SLAM-ER) was a major modification of the SLAM (which in turn was an adaptation of the Harpoon anti-ship missile), intended to give the Navy a standoff capability against land as well as ship targets. Like the Harpoon and SLAM, the SLAM-ER’s prime contractor was McDonnell Douglas (later bought by Boeing). The SLAM-ER had approximately a two-year head start on JASSM with an EMD contract awarded in March 1995. The SLAM-ER had a new airframe/wing design to increase range, a warhead with increased lethality, as well as avionics/software/mission planning upgrades to improve accuracy and make employment easier. As an upgrade to a small number of existing missiles, the SLAM-ER was not subject to the standard acquisition milestone process.
The JASSM and SLAM-ER share many capabilities, but with some important differences. Characteristics of the two missiles are included in Table 2.
The JASSM is heavier, has longer range, and carries a larger penetrating warhead. Guidance systems are similar, with Global Position System (GPS) bringing the missiles close to their targets while imaging infrared sensors are used in the terminal phase. Both missiles use the same Williams turbojet engine. A distinguishing attribute of the SLAM-ER is its two-way data-link with man-in-the-loop functionality. This gives the SLAM-ER the capability to attack moving targets such as ships, as well as providing additional tactical flexibility. The JASSM’s “fire and forget” capability is meant only for stationary targets; a similar capability was included in the SLAM-ER as a retrofit in the FY 1999 production lot. The JASSM is also distinguished by its stealth capabilities.
The JASSM program was required to complete an AoA-like activity prior to Milestone I and the start of the Program Definition and Risk Reduction (PDRR) phase. COEA I compared potential JASSM capabilities and associated technologies to those achievable through modification of existing systems. It was not clear from available documentation whether SLAM-ER was included in COEA I. COEA I found JSSAM the preferred alternative. Milestone I occurred in June 1996, with Lockheed Martin and McDonnell Douglas chosen to design and build prototype missiles. Prior to Milestone II and the beginning of EMD, an updated COEA (COEA II) was required, in which the two candidate systems from the PDRR phase were compared directly to the SLAM-ER (JASSM SAMP, 1997). COEA II also found the JASSM to be the preferred system. Milestone II occurred in November 1998, with Lockheed Martin chosen as the prime contractor.
JASSM acquisition initiatives included adoption of commercial practices, minimization of military specifications and data reporting, and cost as an independent variable. The general approach was to give the contractors maximum flexibility in making trade-offs within the constraints of high-level key performance parameters (KPPs) and a unit cost goal of $400,000–$700,000 in FY 1995 dollars. Another aspect of the acquisition strategy was the use of price-based acquisition (PBA). This meant the inclusion of fixed price options for the first five production lots (accounting for 1,146 of the 2,400 Milestone II requirements) as part of the EMD contract and the elimination of cost reporting for those lots. This resulted in concessionary prices on the part of Lockheed Martin for those lots along with a high risk that subsequent lots would increase in price and with the government left with limited information for use in negotiation and program planning. This problem opened up another opportunity for contrasting JASSM and SLAM-ER.
As a fallout of the PBA strategy, IDA performed an “independent market survey” analysis in support of the JASSM’s 2004 Milestone III full rate production decision (Woolsey et al., 2004). A unique aspect of this was the use of effectiveness analysis to help determine fair prices for the JASSM in relation to the prices and capabilities of other standoff missiles, including the SLAM-ER. The campaign model used did not force the one-for-one substitution of the competitive missiles for JASSMs; instead, platform/weapon/target assignments were determined by an optimization model where Blue (friendly) losses were minimized. From this, a utility measure was specified, Uk=Qjassm/Qk, where Uk was the utility for the kth competitive missile, Qjassm was the quantity of JASSMs planned at Milestone III,2 and Qk was the quantity of the kth competitive missile required for Blue attrition and campaign length to equal those for the JASSM quantities. The utility measure provided a basis for comparing near-substitutable systems. Target prices for the JASSM in relation to the competitors were then defined as = Pk/Uk; for the JASSM to be a “good deal” relative to competitors, its purchase price would need to be equal to or below. In practice Pk/Uk varied over a range, depending on effectiveness modeling assumptions and pricing ground rules for the competitive missiles. In the analyses, the missiles resulting in the lowest Pk/Uk metrics were the SLAM-ER and the Storm Shadow, a France/UK joint venture.
It was never likely that the Air Force would buy the SLAM-ER (Navy News & Undersea Technology, 1999).3 However, there was a possibility that the Navy would procure fewer SLAM-ERs in favor of JASSM. It should be noted that although the Navy was a participant in the JASSM program, its contributions to development were minimal and no procurement was funded (although quantities up to around 700 were considered). In the end, the Navy did not significantly change their SLAM-ER inventory goal (which varied between 400 and 600), although they did buy out their requirement substantially faster than originally planned. The Navy formally pulled out of the JASSM program in February 2005; they were satisfied with the SLAM-ER for their standoff missile requirements (Fein, 2005).
The JASSM was subject to a Nunn-McCurdy breach in April 2007, primarily because of increases in procurement unit costs. The program was not recertified until one year later; the delay was mainly because of concerns regarding reliability. During this period, the Air Force released a Request for Information for alternative missiles. We do not know from available documentation whether Boeing responded with the SLAMER (Putrich, 2008).
The SLAM-ER and JASSM exerted competitive pressure on one another throughout their acquisition cycles. However, following Milestone II, decision makers never set up a “do or die” moment for either system, as was the case for the C-17/NDAA. It is not clear whether program outcomes were materially affected. The JASSM program’s ambitious unit price goals may have been partially prompted by competitive pressure from the SLAM-ER; the lower JASSM price estimates were an important advantage in the Milestone II AoA (JASSM AoA, 1998). However, in the course of program execution, JASSM average procurement unit prices almost doubled ($720K versus $400K FY 1995 dollars), while SLAM-ER prices changed little. In terms of missile capabilities, the upgrade paths of the missiles showed convergence in some objective capabilities. Already mentioned is the upgrade of SLAM-ER with automatic target recognition; there are plans for JASSM to add a two-way data link and maritime attack capabilities. JASSM and SLAM-ER compete with one another for foreign sales, although the additional capabilities of JASSM mean that it is available to fewer nations. A notable direct competition was to equip Australia’s F/A-18s, where the JASSM was chosen over the SLAM-ER (Australian Defence Magazine, 2007).
The KC-X Competition
The KC-X tanker aircraft program, an important instance in which near-substitutable systems were part of a formal direct competition, was meant to replace the aging KC-135 fleet of air-refueling tankers. This program also has a unique back-story. The Air Force originally proposed to lease Boeing KC-767 aircraft for use as tankers under a commercial operating lease. Due to shortcomings in that program, the DoD Inspector General (Department of Defense Office of the Inspector General. 2004) recommended that a new acquisition program be implemented that complied with all requirements for a major acquisition program, including the performance of an AoA.
The AoA was performed by RAND, which found that new medium-to-large-sized wide-body commercial aircraft-based tankers (767-747) were the best solution for KC135 recapitalization. Although cost-effectiveness analyses were performed on individual aircraft models within this category, RAND did not think that the cost analyses had sufficient fidelity to determine a specific solution. The price of the “green” aircraft (the commercial aircraft prior to modification to tanker configuration) was cited as an important source of uncertainty. Given this, RAND recommended an open competition to determine the best alternative (Kennedy, 2006).
The Air Force released an RFP in January 2007 for EMD and initial procurement. The offerors were Boeing, with the KC-767, and Northrop Grumman, with the KC-45 (an evolution of the EADS/Airbus A330 Multi-role Tanker Transport [or MRTT]). Both competitors had sold precursor aircraft to foreign customers in limited quantities, but both competitors would need to perform additional development to meet U.S. Air Force requirements. Table 3 presents characteristics for the KC-X precursor aircraft together with the KC-135.
The KC-45 is a substantially larger aircraft than the KC-767. The metric in the RFP most relevant to size was fuel offload/range performance; the offerors were expected to meet or exceed KC-135 performance—the so-called KC-135 KPP. However, the RFP did not indicate that any consideration would be given to by how much it was exceeded.
Another RFP metric relevant to size was the integrated fleet aerial refueling assessment (IFARA) factor. The IFARA factor was derived using a modeling and simulation tool applied to various scenarios; the factor is the inverse of the quantity of an offeror’s aircraft that would be required in order to perform the scenarios divided by the number of KC-135R aircraft needed. This is similar to the type of analysis used in the RAND AoA. The IFARA metric should be positively correlated to maximum fuel weight, but would also take into account other constraints on employing tankers, many of which would advantage a smaller aircraft. The IFARA factor was 1.90 for the KC-45 and 1.72 for the KC-767. Although the IFARA factor could be used as a discriminator, its weighting was low relative to other categories.
The Air Force selection board chose the Northrop Grumman/EADS KC-45. Boeing protested the selection and the protest was sustained on a variety of counts (General Accountability Office, 2008). The most important was the use of performance above the KC-135 KPP as a discriminator in violation of the RFP instructions.
The problems with the KC-X program demonstrate the complications associated with a direct competition between near-substitutes. In the course of direct competition, the weighting of discriminators that flow from divergent attributes will be critical. The use of modeling and simulation tools that can aggregate over different attributes to form discriminators in a single dimension is a possible response. Although the IFARA factor was consistent with this approach, it was given minimal weight in the original competition. However, for the selection criteria included in the revised KC-X RFP (Department of the Air Force, Air Force Materiel Command, 2010) the IFARA factor was more central, as it was included as an adjustment factor on the offerors’ pricing data. Boeing was declared the winner of the second competition. Although the ultimate success of the KC-X program has yet to be determined, it is likely that the prices originally established as part of the sole-source lease will be improved upon.4
In general, the purpose of competition between near-substitutes has not been to gain lower prices, but has focused on capability and quality issues. Direct competition in which the offered price is the key metric has been the exception. Almost by definition, the up-front investment for competition between near-substitutes is relatively small as the systems/platforms will likely already exist in some form (this was the case for all of our case studies). The primary benefit is to provide decision makers with multiple ways to fill capability gaps. The best way (and perhaps the only valid way) to determine the relative value of near-substitutes is by employing cost-effectiveness analyses that portray the different interactions of the near-substitutes with complementary military force structure and physical infrastructure. This clearly is a difficult proposition in the case of direct competition, although the second KC-X RFP attempted to implement this.
In terms of the industrial base, near-substitutes provide the opportunity to expand the base of suppliers. This can be seen in the example of Boeing commercial in the C-17/NDAA case and EADS/Airbus in the KC-X case. It is interesting to note that the acquisition of McDonnell Douglas by Boeing would now hamper the participation of Boeing commercial as an alternative supplier of airlift aircraft, given the C-17’s incumbent status.
How might the consideration of near-substitute systems to meet military requirements be encouraged? The requirement for an AoA is certainly a good starting place. However, in current acquisition policy, an AoA or a similar cost-effectiveness analysis is not required after Milestone B (formerly Milestone II). In the C-17/NDAA example, a post-Milestone B cost-effectiveness analysis was undertaken because of pressures from the Congress and DoD leadership. In the JASSM/SLAM-ER case, no follow-on comparative cost-effectiveness analysis was performed even though the pre-Milestone II COEA reflected relative pricing assumptions that changed substantially as the program progressed. One way to encourage competition would be through a “rolling cost-effective analysis” process, where the original AoAs would be updated as material changes become evident in the program or potential near-substitutes. As a program office itself may be in a de facto advocacy position, any action encouraging competition from a near-substitutable system may need to be initiated by an outside authority.
Air Force won’t use Navy missile in Kosovo. (1999, April 12). Navy News & Undersea Technology.
ADF weapons: was JASSM the right choice? (2007, August). Australian Defence Magazine.
Bolkcom, C. (2007). Military airlift: C-17 aircraft program. Washington, DC: Congressional Research Service.
Decision, Matter of The Boeing Company, B-311344 et al. (June 18, 2008). United States General Accountability Office.
Department of Defense Office of the Inspector General. (2004). Acquisition of the Boeing KC-767A tanker aircraft. Washington, DC.
Department of the Air Force, Air Force Materiel Command. (2010, February 24). KC-X tanker modernization program, solicitation number: FA8625-10-R-6600.
Dominy, J. R. et al. (2012). The mechanisms and value of competition for major weapon systems (IDA Paper P-4678). Alexandria, VA: The Institute for Defense Analyses.
Fein, G. (2005, June 17). Services need to pursue a common missile, admiral says. Retrieved from Defense Daily: http://www.defensedaily.com/articles/dd/2005/dd06170502.html
Forecast International. Retrieved from http://www.forecastinternational.com/
Greer, W. L. et al. (1993). Cost and operational effectiveness analysis of the C-17 program (IDA Report R-390). Alexandria, VA: Institute for Defense Analyses.
Joint Air-to-Surface Standoff Missile (JASSM) Analysis of Alternatives, Appendix G cost analyses (U). (September 1998).
Joint Air-To-Surface Standoff Missile (JASSM) Single Acquisition Management Plan (SAMP) for Milestone II, Revision 1. (December 1997).
Kennedy, M. (2006). Analysis of Alternatives (AoA) for KC-135 recapitalization, executive summary. Santa Monica, CA: RAND Corporation.
Putrich, G. S. (2008, May 2). DoD approves Air Force’s JASSM. Retrieved from Defense News: http://www.defensenews.com/story.php?i=3510334
Rogerson, W. P. (1994). Economic incentives and the Defense procurement process. Journal of Economic Perspectives, 8(4), 65–90.
United States General Accounting Office. (1994, April 15). C-17 settlement is not a good deal. Washington, DC.
Woolsey, J. P., Frazier, T. P., Harmon et al. (2004). Joint Air-to-Surface Standoff Missile (JASSM) market survey (U) (IDA Paper P-3824). Alexandria, VA: The Institute for Defense Analyses.
1This is consistent with DoDD 5000.1, which states “Acquisition managers shall take all necessary actions to promote a competitive environment, including the consideration of alternative systems to meet stated mission needs.”
2At this point, the planned quantity had increased to 4,250.
3The Navy had offered 40-50 SLAM-ERs to the Air Force for use on B-52s against Yugoslavia, with the Air Force refusing the offer; reports indicated that the Air Force viewed the SLAM-ER as a threat to JASSM procurement.
4Unit price data associated with Boeing’s winning bid have not been released.
Bruce Harmon is an adjunct research staff member at the Institute for Defense Analyses, where he has worked for over 25 years. Bruce has extensive experience modeling the costs and schedules of various aerospace systems, as well as analyses of other acquisition issues. He is a Ph.D. Candidate in Economics at The American University, Washington DC.