By James P. Woolsey, Harold S. Balaban, Kristen M. Guerrera, & Bruce R. Harmon
Typically dual-source competition involves more than one producer building the same build-to-print (BTP) hardware design. A more analytically challenging case occurs when two contractors develop and produce two different designs to meet the same functional requirements. Such a case was examined by the Institute for Defense Analyses in a forward-looking cost and economic analysis of the Joint Strike Fighter (JSF) alternative engine program.
We first considered the additional costs required to execute a competitive program between the F135 (Pratt & Whitney) and F136 (General Electric) engines. While the start-up costs (in the JSF case, an additional development program) and loss-of-learning and rate effects associated with an additional production line are analogous to those in a BTP competition, there is an array of other costs associated with the support of an additional engine design. These were accounted for in IDA’s analyses, including additional government management, initial spares, and depot/training/support equipment. Also included were additional operations and support (O&S) costs such as depot-level reparables, sustaining engineering, and contractor program management; software support, and engine component improvement programs.
All of these costs were considered as the investment required to establish competition. To have the potential for recovering this investment over the JSF’s life cycle, both procurement and contractor-provided O&S services would have had to be competed effectively, and such a competition would have had to save about 18 percent of total procurement and O&S cost.
This paper describes a forward-looking cost and economic analysis of the JSF alternative engine program performed by the Institute for Defense Analyses.
Introduction and Approach
Typically, dual-source competition involves more than one producer building the same build-to-print hardware design. A more analytically challenging case occurs when two contractors develop and produce two different designs to meet the same functional requirements. An example of this type of competition was examined by the Institute for Defense Analyses (IDA) in a forward-looking cost and economic analysis of the Joint Strike Fighter (JSF) alternative engine program. This paper shows that in such a case, a broader range of costs and other factors must be considered when describing the probable effects of competition.
The John Warner Defense Authorization Act for Fiscal Year 2007 directed the Secretary of Defense to select a Federally Funded Research and Development Center to conduct an independent cost analysis of the JSF engine program. The Office of the Under Secretary of Defense (Acquisition, Technology and Logistics) selected IDA to perform that study. This paper highlights some of the more analytically interesting aspects of the study while describing a prototype for similar future studies. The full study is documented in Woolsey et al. (2007).
The JSF engine program had been structured and executed to allow effective competition between two engines, the F135 (being developed by Pratt & Whitney [P&W]) and the F136 (developed by the Fighter Engine Team [FET] of General Electric [GE] and Rolls Royce until cancellation in 2011). The engines were designed to be physically and functionally interchangeable, giving the federal government flexibility in its selections. The planned production quantities were high enough (over 3000 installed engines) that half of the planned purchase would represent a large production quantity to either contractor team. Although the F135 had a head start in development and planned production relative to the F136, the differences were not significant enough to saddle the F136 with a competitive disadvantage.
Past experiences with engine competitions led to the expectation that the two engines would be price-competitive, an important ingredient in a successful competition. These programmatic characteristics are necessary for competition, but are an insufficient basis for a decision about whether a competitive program would benefit the government. Our analysis examined the costs and potential benefits of the sole-source and competitive approaches to providing engines for the JSF program.
In approaching the problem, we first considered the investments required to execute a competitive engine program. We then determined the savings that would have to be achieved as a result of the competition to recover this investment and compared these savings to what has been seen in other competitive programs. Finally, we evaluated potential benefits of competition beyond price reductions. The ground rules and assumptions used for the analyses were as follows:
- Analysis was for unique components only (no lift fan, nozzle, or roll posts that would be shared across both designs)
- Procurement profiles for US and international partners were from the 2006 JSF Selected Acquisition Report
- Analysis does not include costs and benefits
- to international customers or future US
- Costs through FY 2007 were considered sunk
- JSF program office ground rules provided the baseline for Operations and Support (O&S) cost analysis
- The life-cycle period was 2008–2065
Calculations were performed on a net present value (NPV) basis as well as in constant fiscal year 2006 dollars (FY06$) and then year dollars (TY$). In addition to a baseline case reflecting the program of record, multiple excursions were also considered.
Investments Required for a Second Engine Program
Execution of a second engine program requires additional costs in all phases of the program life cycle. These investments include both direct investments, such as development costs for the second engine, and opportunity costs, such as the loss of economies inherent in larger production quantities. These additional costs would be evident in support activities as well as in the traditional investment phase of the acquisition cycle. A major challenge was enumerating all aspects of cost touched by the existence of a second engine design.
For System Design and Development (SDD) to be accomplished, the F136 design needed to be completed; it would have had to go through both ground and flight testing; and it would have had to be managed and integrated into the JSF system by Lockheed Martin, Federal government, and P&W personnel. Costs were estimated by IDA for the following activities:
- Remaining F136 FET SDD contract costs
- Government ground test
- Government and Lockheed Martin personnel supporting F136 development
- Fuel and Other Remaining P&W support to the FET
The most important part of the remaining F-136 development cost was the FET SDD contract. We examined the schedules and associated budget plan for both the F135 and F136 SDD contracts using a model developed from historical data. In both cases, we found a substantial risk that the Initial Service Release milestone for each engine would be delayed for around one year; given this, a higher cost estimate for the F136 was used as an excursion in the NPV analyses.
During production, the presence of a second engine would reduce the quantities P&W produced, consequently reducing the economies that would have accompanied a larger purchase. Scale economy effects included both cost progress (learning) and production rate impacts. The existence of a second engine also impacts procurement costs through the need to establish support infrastructure for the second engine design.
Quantifying production cost differences meant creating independent cost estimates for the F135 and F136. As the F135 was in a more advanced state of development, data were available describing the costs of manufacturing flight test engines. We used F135 Flight Test Engine (FTE) #3 actual data at the component level. Component learning curves from P&W’s F119 (the F-22 engine that was a precursor to the F135) FTE and production experience were applied. The few components that were common between the F119 and F135 shared learning quantities. The F136 was more challenging, as no meaningful test hardware costs were available. Instead, we developed component cost estimating relationships (CERs) from cost data for previous GE engines (the F101, F110, F404, and F414); components modeled were the fan, core, low-pressure turbine, augmentor, and final assembly/other. F136-specific design data for each component were used along with historic GE price-level learning curves to produce F136 production costs.
The resulting cost estimates reflected a sole-source environment for both engines. Given these cost estimates and the timing of production schedules, including “education buys” for the F136 (early procurement lots not under competition), we found the cost disadvantage of the F136 at the first competitive lot to be smaller than those calculated for earlier successful competition programs.
Once we had learning curves for each engine, calculating the loss of learning associated with a split buy compared to an F135 sole-source program was a straightforward exercise. To do this, we assumed a 50/50 split of production engines between the two models. Note that in this calculation, no competition effects were taken into account—the learning curves were based on sole-source experience. The impact of competition on learning curves is taken into account later in the analyses.
Another cost difference considered was the change in overhead costs paid by the government, given a second engine producer. Moving engines from the P&W to FET facilities would affect total overhead costs paid by the US government (including programs other than the JSF); we modeled this effect by assuming:
- 50 percent of total costs were overhead,
- 30 percent of overhead was fixed, based on defense aerospace averages, and
- Effects at GE facilities also applied to Rolls Royce content.
Business base projections came from public data. The analysis showed an increase in overhead cost for dual-sourcing the JSF engine of $228 million (FY06$) over the period 2006–2034. This may modestly overstate the effect because some fixed overhead effects were captured in the learning curve analysis (learning curve slopes would be marginally shallower if fixed costs were excluded in their calculation).
The cost of initial spare parts and establishing repair depot capabilities would also be increased by the introduction of a second engine. For initial spares, a second engine program creates higher spares cost because of higher unit engine production costs and the requirement for two spares pools. The costs of depot establishment for a second engine were based on F119 cost experience and contractor, program office, and IDA estimates from previous studies. These estimates were adjusted for engine quantity, number of depot locations, and engine attributes.
Operations and Support (O&S)
In the support phase of the system’s life cycle, the second engine would also increase the costs of supporting the JSF engine inventory. As was the case for procurement, costs were estimated for both a single engine fleet and a fleet split evenly between the two engine designs. For presentation, we broke the costs into three categories: (1) variable O&S [costs that vary with the number of flying hours], (2) fixed O&S [those that do not vary based on activity], and (3) component improvement programs [CIP]. The CIPs are not strictly O&S costs, as they are paid for by Research, Development, Test & Evaluation (RDT&E) funding; however, they are oriented to solving in-service problems.
The most important variable O&S costs are depot-level reparables (DLR) and consumables. The drivers of DLR/consumables costs were reliability and repair costs. IDA used a DLR CER and data at the engine module level to estimate the total DLR and consumables costs. The relationships were calibrated using data from the JSF program office, the Air Force, and historical data for the F-15 and F-16 programs. The analysis accounted for reliability growth as well as the effects of aging and diminished parts supplier sources. DLR/consumables costs were higher for the two-engine case because maturity with respect to reliability was reached two years later, and repair costs (including consumables) tended to scale with unit production costs, which in turn were higher for the two-engine case.
Fixed O&S costs included Sustaining Engineering/Program Management (SE/PM) and post-deployment software support. SE/PM annual fixed costs were based on F-119 SE/PM experience adjusted for engine complexity, configuration, and program scale. Software support was estimated using the Constructive Cost Model (COCOMO) maintenance model structure, with estimates driven by Source Lines of Code (SLOC), SLOC change and growth rate, productivity, and labor rates. Fixed O&S costs essentially double with the second engine.
Statistical analyses of historical annual CIP funding found three important cost drivers:
- The size of the engine inventory—the larger the inventory, the greater the payoff for a given upgrade;
- Complexity and size of the engine being supported—engines that are costlier to build are generally costlier to improve; and
- Time trend effects—as engine development practices improve, CIP costs decrease, and as individual engine models mature, CIP requirements decrease.
Average annual CIP funding was estimated at $26 million (FY06$) per engine type; peak funding of $40 million per engine type would occur in FY 2016. Because of the inventory effects, the two-engine CIP costs were slightly less than twice the sole source F135 case.
Table 1 presents the O&S costs for the sole source purchase and a 50/50 split, as well as the delta associated with adding the F135 engine.
Note: Values do not add due to rounding
a Other includes maintenance manpower, modifications, contractor logistics support, and indirect support
Our estimate of the sum of these investments, including opportunity costs, is $8.8 billion in constant FY 2006 dollars, before accounting for the price reductions that competition might produce. Of this investment, we estimate that $2.1 billion would occur in fiscal years 2008–2012. This is primarily development cost, while the residual amount (total less O&S and 2008–2012 costs) of $2.1 billion is mostly procurement cost.
Potential Price Benefits from Competition
We examined the history of two-engine competitions—the so-called Great Engine War (GEW) and the dual-sourcing of the F404 engine. We also surveyed past studies of competition savings.
As in most studies of competition savings, the key analytical tool applied was the learning/price improvement curve. Figure 1 presents a generic example of its application to calculate competition savings.
In our application, competition savings percentages are the percentage displacement of the observed competition cost data from the projected sole-source price improvement curve. This displacement corresponds to the gross unit price savings in Figure 1. Note that the loss-of-learning costs shown above were accounted for in our analyses as investment costs. Past study results could only be useful for reference if the gross savings percentages were calculated in a consistent fashion and the calculations were transparent.
The GEW was the US Air Force-initiated competition in 1984 between P&W and GE for 2000 F-15 and F-16 fighter engines. Analyses of competition savings for the GEW were complicated by the specifics of the engine programs. This led us to two different approaches for estimating competition savings. The P&W F100-100/200 was the incumbent engine in the F-15 and F-16 programs. Parallel with the start of competition was the introduction of a substantially upgraded version of that engine, the F100-220. Given this, direct comparisons between the competition prices and the sole-source price improvement curve could not be made. The introduction of new components caused the F100-220 to experience a substantial loss of learning with a retracing of the original F100-100/200 price improvement curve. When we adjusted for this, we found prices for the normalized F100-220 to be significantly below those for the sole-source F100-100/200. An alternative approach was to compare prices for GE’s entry—the F110-100—with those derived from the sole source F100-100/200 learning curve; this methodology also showed significant price savings.
The F404 engine competition was a dual-sourcing of engines used on US Navy F/A-18 aircraft. In this case, P&W built engines to the GE F404 design and competed in four competitive procurement years, 1986 through 1989. The competition was subsequently terminated. The approach used in estimating savings followed closely the generic methodology presented in Figure 1.
Our analyses showed gross savings ranging from 11 to 18 percent of engine procurement costs. We also examined available studies of competitions that the Department of Defense (DOD) conducted over the past 30 years for a variety of other systems. Unfortunately, we were not able to extract results from them that could be used as a primary means of evaluating the likely savings of a JSF engine competition. There were methodological differences among the studies, and some studies did not describe clearly how the analyses were performed and what was included in the stated “savings due to competition.”
IDA employed a discount rate of 3.00 percent, based on 30-year maturity in Appendix C of Office of Management and Budget Circular A-94; this translated the total investment of $8.8 billion in FY 2006 dollars into an NPV of $5.1 billion. To break even financially—or, in other words, to offset fully the NPV of investment estimated to establish the alternative JSF engine—would require a savings rate during the production phase of 40 percent. Savings of this magnitude are implausible, considering the 11 to 18 percent savings realized in previous engine competitions. If O&S costs were effectively competed in addition to procurement costs, the required savings rate would fall from 40 percent of procurement costs to 18 percent of total costs. Excursions reflecting other likely scenarios did not change the break-even savings percentage appreciably. For example, higher F136 SDD costs associated with a schedule stretch resulted in a break-even percentage savings of 19 percent, while a 50 percent increase in buy quantities resulted in a break-even percentage savings of 16 percent.
Because the DoD has not typically linked procurement and O&S costs in a single competition, we found no historical data with which to estimate plausible O&S savings under such an acquisition strategy. Competition might affect prices for O&S services in a range of ways. Without explicitly competing support services, some O&S savings would flow naturally from the savings in a procurement competition. Spare parts, for example, could be expected to see some savings through this mechanism. Elements of O&S can also be tied to the procurement competition by adding O&S metrics to the procurement selection criteria. To take O&S competition a step further, all elements of O&S services could be packaged into a single acquisition covering design improvements, spare parts, and logistics support. This model is widely used by the commercial airline industry, which routinely bundles support contracts with the initial engine purchases, bringing support services directly into the purchase competition. We understood at the time that the JSF program office intended to use an acquisition strategy that ties some elements of O&S costs to the procurement competition.
Other Benefits of Competition
Competition had the potential to bring benefits in addition to price reductions. One such potential benefit was fleet readiness. The JSF will dominate the US fighter attack force structure as no previous platform has. Having two independent engine types could reduce the impact of an engine anomaly that could ground or reduce the readiness of large numbers of aircraft. Also, competition might have improved contractor responsiveness. For example, it is generally agreed that the government received improved contract terms, cooperation, and overall responsiveness from contractors when competition for fighter engines was introduced during the 1980s.
Finally, continuation of the F136 program would have ensured that GE would remain in the industrial base for high-performance military aircraft engines. Without the F136, GE’s incentive and ability to maintain the skills unique to these types of engines is less certain, although GE would remain a leading supplier of commercial aircraft engines.
Creating competition by developing, procuring, and maintaining a second engine would require an investment of about $8.8 billion in constant FY 2006 dollars. Approximately half of these costs would occur in the operations phase of the program. To have the potential for recovering this investment over the JSF’s life cycle, both procurement and O&S services would have to be competed effectively, and such a competition would have to save about 18 percent of total procurement and O&S cost. The study results show the importance of a comprehensive accounting of all costs that would be affected by the introduction of a second hardware design as part of a competitive strategy.
Woolsey, J. et al. (2007). (U) Joint strike fighter (JSF) engine cost analysis: final report (IDA Paper P-4232). Alexandria, VA: Institute for Defense Analyses. Unclassified (PI/LR/FOUO).
Jim Woolsey is the deputy director – Performance Assessment, AT&L, Performance Assessment and Root Cause Analysis. He is on an Intergovernmental Personnel Act assignment from the Institute for Defense Analyses (IDA), where he is an assistant director of the Cost Analysis and Research Division. Prior to his employment at IDA he was lead F/A18C/D structures engineer, Naval Air Systems Command. Mr. Woolsey has a BS in Aerospace Engineering from the Virginia Polytechnic Institute and State University and an MBA from George Mason University.
Harold Balaban is a research staff member in the Cost Analysis and Research Division of IDA. He specializes in applying reliability, maintainability, and logistics factors to estimate the life cycle cost of major defense systems. Prior to his employment at IDA, Dr. Balaban was Director of Advanced Analysis at ARINC Research Corporation, Annapolis, Maryland. He has a Ph.D. degree in statistics from The George Washington University.
Kristen Guerrera is a research associate in the Cost Analysis and Research Division at IDA. Her work at IDA has supported numerous independent life cycle cost estimates and analyses of alternatives, with a focus on operation and support costs and modeling and simulation. She has also performed numerous analyses for the Department of Veterans Affairs including an assessment of VBA’s quality program, an analysis of the variance in disability compensation across states, an assessment of personnel requirements, and several workload forecast modeling studies. She has a B.S. degree in Industrial and Systems Engineering from Binghamton University and an M.S. degree in Operations Research and Industrial Engineering from The Pennsylvania State University.
Bruce Harmon is an adjunct research staff member at IDA where he has worked for over 25 years. Bruce has extensive experience modeling the costs and schedules of various aerospace systems, as well as analyses of other acquisition issues. He is a Ph.D. Candidate in Economics at American University, Washington, DC.