To print a PDF version of this article, click here.
A risk-driven contract structure is proposed to enhance the cost realism of competitive proposals for the Engineering and Manufacturing Development (EMD) phase of the acquisition life cycle. The authors employ an economic theory framework to discuss how cost-plus contracts typically used during this phase have inadvertently reinforced the sources of contractor and government optimism bias. By mapping probabilistic cost estimates to profit distributions, risk-driven contracts offer a structured method to expose contractors to more cost risk during EMD. Holding contractors accountable for their cost estimates and cost performance should enhance the realism of cost proposals, limit the government’s ability to commit to too many programs, and reduce the cost growth that continues to plague the defense acquisition system.
The Government Accountability Office (GAO) reported a combined $296 billion in cost growth on the Department of Defense’s 96 major acquisition programs in fiscal year (FY) 2008. Sixty-nine percent (64 of the 96 programs) experienced cost growth, demonstrating that the cost growth is not just limited to a few programs. In addition, 42 percent (40 programs) reported at least 25 percent unit cost growth, demonstrating that the bulk of the growth is not limited to a few programs either. Finally, 75 percent (69 programs) experienced increases in research, development, test, and evaluation (RDT&E) costs, demonstrating that problems often start early in the acquisition life cycle (GAO, 2009, p. 2). This last statistic is particularly important to this research since risk-driven contracts are targeted at improving cost realism for system development efforts.
To put this $296 billion cost growth into perspective, consider that the FY 2012 President’s Budget Request is $671 billion (including funding for the operations in Afghanistan and Iraq), with $204 billion allocated to acquisitions ($128 billion for procurement and $76 billion for RDT&E) (DoD, 2011, p. 8-3). Thus, if DoD still wants these 96 weapon systems, it must cover an unfunded liability greater than its annual acquisitions budget. This daunting task is compounded by the current state of the economy and the resulting fiscal pressures. Defense Secretary Robert M. Gates (2011) remarked:
This department simply cannot risk continuing down the same path–where our investment priorities, bureaucratic habits, and lax attitudes towards costs are increasingly divorced from the real threats of today, the growing perils of tomorrow, and the nation’s grim financial outlook.
In support of enhancing cost realism, this article is organized into three parts: (a) a brief review of the difference between cost growth and cost overruns, (b) a discussion of the primary reasons for unrealistic cost estimates, and (c) a detailed demonstration of risk-driven contracts.
Cost Growth vs. Cost Overruns
Cost growth implies an increase in the life-cycle cost estimate, which may or may not affect the cost performance of the current contract. For example, a choice to use a specific material during system development could lead to increased procurement costs without necessarily increasing the development costs. On the other hand, a cost overrun results when a program exceeds the target cost of its contract, which usually leads to life-cycle cost growth despite the prospect for future efficiencies.
When target costs are unrealistic, overruns do not necessarily indicate excessive expenditures (Cummins, 1977, p. 179). Despite the reasons for overruns, they are almost always counterproductive. First, they often lead to funding instability within a portfolio, which in turn leads to adjustments between programs (damaging healthy programs to rescue sick ones), reductions in requirements or procurement quantities, or extensions to schedules (GAO, 2008, p. 11). Second, overruns can damage public perception and, as a result, diminish congressional support and risk eventual cancellation (Cummins, 1977, p. 179). And third, overruns can be perceived as a managerial failure and lead to drastic personnel replacements in the government and contractor program offices (Scherer, 1964, pp. 275–276).
Reasons for Unrealistic Cost Estimates
Cost estimates can be unrealistic for a multitude of reasons, which include an overemphasis on the technical cost drivers, optimism bias, and misaligned contract incentives.
Overemphasis on Technical Cost Drivers
While room for improvement always exists, today’s professional cost estimators have an abundance of tools from which to leverage best practices. Sophisticated cost-estimation guides have been published by the Army, Navy, Air Force, National Aeronautics and Space Administration (NASA), GAO, RAND, International Society of Parametric Analysts/Society of Cost Estimating and Analysis (ISPA/SCEA), and the Space Systems Cost Analysis Group (SSCAG). Also available are extensive articles, conferences, and training and certification opportunities from professional societies like ISPA, SCEA, SSCAG, and the United Kingdom’s Society of Cost Analysis and Forecasting (SCAF). In addition, Garvey (2000) authored the definitive textbook on cost estimation wherein he describes the principal methods for addressing cost uncertainty. Finally, a vast array of software tools can be used to construct cost estimates, such as the Automated Cost Estimating Integrated Tools (ACEIT), Crystal Ball, @RISK, PRICE, System Evaluation and Estimation of Resources (SEER), NASA/Air Force Cost Model (NAFCOM), Constructive Cost Model (COCOMO) II, and Constructive Systems Engineering Cost Model (COSYSMO). In an unbiased world, subject matter experts applying these tools and best practices would generate more accurate and reliable cost estimates. But the problem is not a lack of guidance or tools—it is that the cost estimation community usually considers only the technical variables contributing to cost risk.
An understated cause of cost overruns is optimism bias, which is defined as the tendency for people to be overconfident in their predictions (Valerdi & Blackburn, 2009). A common form of optimism bias is optimistic technical estimates, which range from the weight of a hardware component to the number of software lines of code. Perhaps the most difficult and subjective part of cost estimation is eliciting these estimates from technical experts. Unfortunately, it has been shown that most experts are overly optimistic in providing both their most likely and worst-case estimates (Russo & Schoemaker, 1992). Hubbard (2010, pp. 57–77), building on the original research of Brier (1950), provides a practical technique to “calibrate” experts to provide better estimates when confronted with uncertainty.
A second, and equally damaging, form of optimism bias is optimistic management estimates by both contractors and the government. The contractor’s optimism bias is caused by pressures to win competitions. William M. Allen, Boeing’s president in 1964, admitted, “I can think of a lot of programs in the Boeing Company where, if the estimate had been realistic, you wouldn’t have had the program. And that is the truth” (Butts & Linton, 2009, p. 36).
While two or more contractors are often funded during early technology development and prototyping efforts, the government typically only funds a single contractor during EMD due to prohibitively high system development costs. After several years of focused government investment, the incumbent contractor normally develops a significant technical advantage. Thus, the government’s options are greatly limited since the prospect of reattempted competition is dubious at best. As a result, the contractor that wins the competitive EMD downselection usually monopolizes the production and sustainment efforts as well. With so much long-term revenue and profit on the line, competition to win the EMD contract is intense. And since cost is a leading variable in the government’s source selection, there is a strong motivation to provide the lowest cost proposal.
The government’s optimism bias is caused by the Services’ desire to secure funding for new programs and sustain funding for existing ones. To maintain the appearance of affordability, cost estimates that fit within authorized budgets are at least tacitly encouraged (Williamson, 1967, p. 229; GAO, 2008, pp. 20-21). In addition, U.S. Senators and Representatives often contribute to the government’s optimism bias by supporting programs with poor business cases when the funding is allocated to their constituents.
Misaligned Contract Incentives
While strong leadership and accountability may help reduce optimism bias amongst stakeholders, properly implemented contract incentives are an even stronger antidote. Figure 1 organizes the most prevalent contract types by their degree of risk sharing and typical use throughout the acquisition life cycle. Cost Plus Fixed Fee (CPFF) and Firm Fixed Price (FFP) contracts represent two polar extremes with no risk sharing. The government assumes all cost risk in a CPFF contract, and the contractor assumes all cost risk in an FFP contract. Cost Plus Incentive Fee (CPIF) and Fixed Price Incentive Firm Target (FPIF) contracts offer a middle ground with risk sharing by both the government and contractor. Of these two incentive contracts, only FPIF contracts expose contractors to a potential loss, but as with FFP contracts, maximum losses are not constrained. Theoretically, a contractor can be forced into bankruptcy in attempting to fulfill the requirements of an FFP contract. However, with the dwindling defense industrial base (Aerospace Industries Association, 2009), it is not in the government’s best interest to force a contractor out of business. In addition, contractors are likely to mount protracted legal battles to protect their interests, which are counterproductive in delivering capability to the warfighter and a poor use of taxpayer resources.
On the other hand, a contractor’s maximum liability for overrunning a typical CPIF contract is no profit. While their short-term stock prices may be impacted, at least four reasons can be set forth to explain why contractors still benefit when they receive no profit (Fox, 1974, pp. 242–243):
- Scientists and engineers are gainfully employed (or hired) and available for future programs.
- Technology competency is accrued, which improves their market position for future government and commercial business.
- Facilities and equipment are maintained and often upgraded at the government’s expense.
- Overhead expenses for other programs (and potential new programs) are slightly reduced by contributions to the overhead pool.
Figure 1. Recommended Contract Types for Each Acquisition Phase
Properly designed incentive contracts address classic moral hazard and adverse selection problems (McAfee & McMillan, 1986, p. 326). Moral hazard is the propensity to act differently when insulated from the risk of a loss. Thus, moral hazard encompasses the propensity for contractors to underestimate competitive program costs and carry excess organizational slack during contract execution when not exposed to a potential loss. Organizational slack is characterized by inefficiently high operating and investment expenses (Williamson, 1967, pp. 224–226). Operating expenses can be reduced through the adoption of lean practices if risk sharing is high enough to overcome the cultural barriers to change. In addition, contractors are likely to allocate their best people to the contracts with the largest potential losses, which can also help reduce operating costs. Conversely, less risk sharing is likely to increase organizational slack in favor of more investment expenses. For example, Scherer (1964, p. 263) identifies the government’s source selection emphasis on the availability of skilled manpower as an encouraging factor in contractors maintaining their workforces at inefficiently high levels.
Adverse selection deals with the government’s imperfect knowledge of the expected cost of each contractor. Williamson (1967, p. 230) boldly states, “It is unquestionably true that the government suffers from an information disadvantage.” Indeed, contractors benefit from locally calibrated parametric cost models, employ the technicians and engineers who will be working on the contract, and have close relationships with key suppliers.
If the government had perfect information (and was free from contractors’ moral hazard), it would award a CPFF contract to what it knew to be the lowest cost contractor to avoid the risk premium of incentive contracts (Samuelson, 1986, p. 1,539). However, since the government does not have perfect information and cannot avoid contractors’ moral hazard, economists reject using cost-plus contracts for competitive source selections (McAfee & McMillan, 1986, p. 327). Instead, economists advocate contracts that expose contractors to a potential loss to solicit their unbiased cost estimates, but for system development efforts with high uncertainty, potential contractor losses need to be appropriately limited. Otherwise, to avoid the extremely high cost risks of fixed-price arrangements, contractors may choose not to bid, which would in turn reduce the competition essential to both guarding against overestimation bias and producing viable warfighter options.
As with the cyclic nature of most acquisition reforms, DoD has oscillated back and forth between its preference for cost-plus and fixed-price contracts. Cancian (1995, pp. 195-196) traced the history of this oscillation over the past several decades. In the 1950s, he noted that cost-plus contracts were the norm. The resulting huge overruns led to a preference for fixed-price Total Package Procurement contracts in the 1960s. When this practice failed due to the high risks contractors were forced to assume, cost-plus contracts resumed their prevalence in the 1970s. Amid perceived procurement “scandals,” DoD again shifted its preference back to fixed-price contracts in the 1980s. Of course this policy failed again for the same reasons, bringing the defense acquisition process to its current phase where cost-plus contracts are again dominant.
It appears the pendulum may be swinging back to fixed-price contracts with recent directives published by a former Under Secretary of Defense for Acquisition, Technology and Logistics (USD[AT&L]) (Carter, 2010, p. 6). However, the guidance on using FPIF contracts focuses on early production contracts (just after Milestone C in Figure 1.). This guidance is a step in the right direction away from the subjective Cost Plus Award Fee (CPAF) contracts that have recently become common during early production, but does not address the misaligned incentive structures typically used during system development when the cost uncertainty is even higher.
Rather than continuing to oscillate back and forth between cost-plus and fixed-price contracts, DoD could benefit from embracing a hybrid, risk-driven contract type for system development. As discussed above, FPIF contracts are inappropriate since they do not constrain the maximum loss potential for contractors. CPIF contracts could be used to expose contractors to a limited loss potential by extending the sharing line into the negative fee region, but in practice this is rarely done since negotiating an arbitrary maximum cost point is extremely difficult. For example, if a contractor submits a point cost estimate of $100 million with no further information, how should the maximum cost point be determined? This process is difficult enough when the minimum fee is positive. Negotiating an arbitrary maximum cost point when a $20 million loss is at stake could be unworkable.
Notional Probabilistic Cost Estimates
By taking advantage of modern probabilistic cost estimates, risk-driven contracts provide a structured method to impose a limited loss potential on contractors. Experience has shown that defense acquisition program cost estimates are often best modeled by the lognormal probability distribution because its right skew accurately reflects the disproportionate chance and magnitude of cost overruns (Department of the Air Force, 2007, p. 96).
Figure 2. Notional Probability Distribution Functions
Two lognormal probability distributions will be used throughout this paper to describe the risk-driven contract structure. Figure 2 shows the probability distribution functions (PDF) of “blue” and “red” probabilistic cost estimates with the same mean but difference variances. The blue cost estimate represents a notional Low-Rate Initial Production (LRIP) proposal, and the red cost estimate represents a notional EMD proposal. Note that the red estimate has both a higher cost risk and opportunity than the blue estimate, as shown by its longer right and left-hand tails, respectively. With less of the design locked down, decisions made on the red EMD program often have a larger marginal cost impact than the relatively minor decisions still pending on the blue LRIP program.
Figure 3. Notional Cumulative Distribution Functions
Figure 3 shows the corresponding cumulative probability distribution functions (CDF) which reveal the confidence level of each possible cost from the notional PDFs. For example, there is an 80 percent chance that the red program will cost $133.1 million or less. Table 1 lists selected confidence levels from Figure 3 that are used in this article. Finally, for the purposes of this discussion, the blue and red cost estimates are assumed to be accurate and unbiased. They bound the possible costs without the influence of any technical estimation errors or optimistic biases.
Table 1. Selected Confidence Levels from Figure 3
Fixed Price Incentive Firm Target Contract Structure
Before describing the risk-driven contract structure, the expected profits from an FPIF contract will be briefly outlined for comparison purposes. Consider the FPIF contract structure shown in Figure 4. The solid magenta profit sharing line is applied to both the blue and red cost estimates portrayed on the right “Probability” axis. The target cost is set to $100 million—the expected cost of both the blue and red programs. A $12 million target profit is set for illustrative purposes. Finally, a 50/50 sharing ratio and 120 percent ceiling are set in accordance with USD(AT&L)’s recommended point of departure (Carter, 2010, p. 6). The point of total assumption (PTA) cost and profit ($116 million and $4 million, respectively) are calculated based on the above variables.
Figure 4. Fixed Price Incentive Firm Target Contract Structure
Note. PTA = Point of Total Assumption.
The expected profit of each program is determined by multiplying the profit at each cost by its corresponding probability and then summing all possibilities. Thus, the blue and red cost estimates are seen as weighting functions on the magenta sharing line. The net result is $10.9 million for the blue program and $7.5 million for the red program. Since the expected profits are different for each program, this contract structure is not universally applicable to all cost estimates. To match the expected profits for both cost estimates, a trial-and-error method adjusting the sharing ratios and ceiling percentages would be required.
Figure 5. Probability Domain Representation of Figure 4
Next, observing from Figure 3 that each cost has a corresponding confidence level, it is possible to display the profit sharing relationships in the probability domain, as shown in Figure 5. The blue and red cost estimates each have distinct profit sharing curves. As previously discussed, the red program is seen to have a higher profit opportunity, but also a much higher potential loss. Assuming the cost estimates accurately bound the possible costs (and setting the maximum costs to the 99 percent confidence levels), the maximum loss is $43.1 million for the blue program and $148.4 million for the red program. It must be noted, however, that there is only a 1 percent chance of incurring these maximum losses. At this point, it should be obvious that this FPIF contract structure favors the blue cost estimate. While contractors might agree to this FPIF contract for the blue program, it is highly unlikely they would expose themselves to a $148.4 million loss on the red program even when there is a $7.5 million expected profit.
Risk Aversion in Human Decision Making
Economists have studied the risk aversion propensity of contractors to sacrifice higher expected profit margins in order to minimize their share of potential losses when faced with uncertainty. Scherer (1964, p. 276) collected strong empirical evidence to support this violation of expected profit maximization theory whereby risk-neutral contractors would prefer the contract offering the highest expected profit despite its potential losses. In addition, Kahneman won the Nobel Prize in Economics for modeling the psychology of decision making under uncertainty. Working together with Tversky, Kahneman (1984) confirmed that it is human nature to be risk averse. Their findings support the conclusion that in general people are more likely to settle for a sure gain than gamble for a higher expected gain. For example, most people would rather settle for an $800 sure gain than bet on an 85 percent chance to win $1,000 (with a 15 percent chance to win nothing) even though the latter has the higher mathematical expectation of $850 (Kahneman & Tversky, 1984, p. 341).
Risk-Driven Contract Structure
It should be no surprise that the FPIF example cited previously favors the blue cost estimate, which is more representative of an LRIP program. In addition, the very large potential loss for the red program confirms why FPIF contracts are not typically appropriate for system development efforts during EMD. However, rather than settling for a cost-plus contract variant during EMD, government acquisition officials could benefit from considering a risk-driven contract.
Unlike the FPIF contract structure, which draws sharing lines in the cost domain, the risk-driven contract structure starts in the probability domain, as shown in Figure 6. This illustrative contract is structured by setting four profit points:
- Profit (p25) = $20M
- Profit (p50) = $12M
- Profit (p80) = $0M
- Profit (p95) = -$20M
For example, the target profit is set to $12 million at both the blue and red 50 percent confidence levels. More importantly, notice how determining the zero and $20 million loss levels in the probability domain provides a structured approach to holding contractors accountable for overly optimistic cost estimates or poor cost performance. The sharing lines simply connect (or extend) the profit points, and are again magenta since they apply to both the blue and red cost estimates.
Figure 6. Risk-Driven Contract Structure
By determining profits in the probability domain, risk-driven contracts reward (or penalize) contractors equally for equivalent cost savings effort. For example, reducing costs from the 50 to 45 percent confidence level earns the same profit increase for both the blue and red programs. Thus, risk-driven contracts normalize the relative value of decisions made on programs with different cost uncertainties. This is contrasted with the FPIF contract structure where saving the same dollar amount on either the blue or red program always earns the same profit increase regardless of the amount of effort required to achieve the savings.
Under the risk-driven contract structure shown in Figure 6, the expected profit for both the blue and red programs is $9.5 million. Note that there is no need to adjust sharing ratios or ceiling percentages to achieve the same expected profit as described above for FPIF contracts. In this way, risk-driven contracts could provide a more universal point of departure for EMD contracts. Policymakers would simply have to determine a few profit points in the probability domain as outlined above.
Figure 6 also reveals the same maximum loss for both the blue and red programs. There is a one percent chance that either program might incur a $25.3 million loss. Further, there is only a 20 percent chance of incurring any loss. Again, while the goal is not to set any specific profit or loss policies, it should be noted how the risk-driven contract provides a method to more reasonably limit the potential losses of contractors engaging in risky development efforts. The objective is to set the loss probability and magnitude to the lowest possible levels that will counteract the previously described moral hazard and adverse selection problems.
Figure 7. Cost Domain Representation of Figure 6
It is also instructive to examine the risk-driven contract structure in the cost domain, as shown in Figure 7. The first major observation is the upper end of the red program’s profit is now less than that of the blue program unlike the FPIF contract example shown in Figure 5. The government shares a larger portion of the red program contractor’s upside profit in return for limiting its potential losses. In effect, the contractor trades slightly less profit opportunity for greatly reduced loss risk, which should be an acceptable trade for a risk-averse contractor. In fact, as shown in Table 2, the maximum profit on the red program has decreased from $51.6 million to $28.0 million while the maximum contractor loss has been reduced from $148.4 million to $25.3 million. In addition, the risk-driven contract offers the red program a higher expected profit, $9.5 million as compared to the $7.5 million offered by the FPIF contract. Thus, contractors should clearly favor similarly structured risk-driven contracts over the FPIF contracts for EMD efforts.
Table 2. Comparison of FPIF and Risk-Driven Contract Profits/Losses
|Blue||Red||Blue & Red|
|Max Profit (p0)||$37.3M||$51.6M||$28.0M|
A second major observation from Figure 7 is the flattening of the sharing curve as the cost uncertainty increases. Indeed, it is appropriate for the government to share a larger portion of the cost risk for requiring greater innovation. However, this natural flattening trend also leads to a potential drawback of the risk-driven contract. As the cost uncertainty increases, the government is forced to allocate more funding to the program. In the case of the red program, the government would have to allocate $243.1 million to cover its share of the contract to the 99 percent confidence level without violating the anti-deficiency laws (which require the government to budget to its full contract liability). The government’s liability could be reduced to a more reasonable $174.5 million by agreeing to terminate the contract at the 95 percent confidence level. However, the contractor’s maximum liability would also be reduced from $25.3 million to $20.0 million. Thus, care must be taken to maintain the contractor’s liability at a sufficient level to still motivate unbiased cost estimates.
Risk-Driven Contract Scenario
The extra funding required to cover the upper end of the risk-driven contract value could be considered the usual cost of overruns. Rather than unknowingly starting a system development effort with an optimistic cost estimate and later dealing with an overrun, the risk-driven contract structure should bring more realism to the initial affordability assessment. For example, consider the following scenario: Two contractors bid $1.9 billion and $2.0 billion for a competitive cost-plus EMD contract. The government’s independent cost estimate is $2.5 billion, so the government awards the $1.9 billion proposal and sets aside an additional $400 million for management reserve. However, 2 years into the 3-year contract, the winning contractor projects an estimate at completion of $3.0 billion. The government is left with two undesirable choices: cancel the program and lose the investment or scramble to find an additional $700 million to cover the overrun.
The scenario just described could be improved through risk-driven contracting. Being exposed to the risk of a loss, the contractors should provide more realistic cost proposals. Perhaps they bid expected costs of $3.0 billion and $3.2 billion. Even more, the cost proposals are probabilistic, giving the government much more visibility into the range of possible costs as opposed to the point estimates normally provided today. Given its $2.5 billion independent cost estimate, the government may be surprised by the high contractor cost estimates and needs to decide whether the weapon system is still worth the expected cost. However, in this case, the knowledge-based affordability assessment is made before the contract is started. And if the contract is still awarded, there is a much better chance it will be adequately funded.
Conclusion and Recommendations
Risk-driven contracts are aimed at reducing cost overruns during the EMD phase of the defense acquisition life cycle. Unlike the traditional cost-plus contracts typically used during this phase, risk-driven contracts offer a structured approach to impose a potential loss on contractors despite the higher technical uncertainty. By exposing contractors to more cost risk, risk-driven contracts should overcome the issues related to moral hazard and adverse selection, and thus motivate contractors to provide more realistic cost estimates and implement more cost control discipline during contract execution. Furthermore, unlike fixed-price contracts where losses are unconstrained, risk-driven contracts appropriately limit potential losses, so competition should not be unduly hindered.
Engineering Change Proposals
To make up for unrealistic initial estimates, contractors often count on ECPs to increase profit margins. Unfortunately, risk-driven contracts do not directly solve this dilemma. However, with increased exposure to losses on the base contract, contractors will likely:
- demand more clearly defined requirements and responsibly limit requirements creep;
- augment precontract planning tasks (such as securing vendor commitments and investing in technical feasibility assessments);
- propose more mature technologies to reduce technical uncertainty; and
- recommend incremental or spiral development strategies.
While these initiatives may help limit the need for downstream changes, the government often adds new contract requirements to keep pace with commercial technology development or evolving warfighter needs. In this case, the government should consider applying ECPs to separate contract line items to avoid disrupting the base contract incentive structure. In addition, the government may want to prenegotiate use of the original probabilistic sharing structure for all ECPs to streamline future contract actions.
Risk-driven contracts should also help limit the government’s ability to commit to too many programs by fostering knowledge-based affordability assessments. By requiring the government to set aside funding to cover the entire contract liability, the anti-deficiency laws should help reduce overextended budgets and the funding instability they induce. The government still reserves the right to deobligate funding from a risk-driven contract in response to changing priorities. However, upsetting the risk-driven sharing ratios will require more negotiation effort than, for example, borrowing money from a CPAF contract. This higher negotiation threshold may provide risk-driven contracts slightly more protection from funding cuts and the resultant schedule delays.
In implementing the Weapon Systems Acquisition Reform Act of 2009, the USD(AT&L) directed program cost estimates to be stated at the 80 percent confidence level (Carter, 2009, p. 6). However, this directive only applies to Office of the Secretary of Defense and Service cost estimates, and not contractor proposals, which normally provide no stated confidence level for their point estimates. To enable risk-driven contracts, the government needs to start requiring probabilistic cost estimates as part of its Request for Proposal instructions. Surprisingly, this is not already common practice, and the government continues to make huge financial commitments without soliciting the confidence level of contractor cost estimates.
Weitzman (1980) states, “The government is frequently assumed to be risk-neutral as a first approximation” (p. 723). Thus, in evaluating probabilistic cost estimates, a risk-neutral program office should generally select the proposal with the lowest expected cost (all other factors being equal). However, given the current fiscal environment and the negative perception caused by overruns, a risk-averse program office may want to also consider the variance of each cost estimate. In other words, it may be prudent to select a proposal with a higher expected cost if it has a lower maximum liability than the other options.
To print a PDF version of this article, click here.
Maj Sean P. Dorey, USAF, is an Air Force Fellow with the Massachusetts Institute of Technology’s Lean Advancement Initiative. He holds a master’s degree in electro-optics from the Air Force Institute of Technology and is Defense Acquisition Workforce Improvement Act-certified in Systems Planning, Research, Development, and Engineering (Level III) and Program Management (Level II). He has served as a deputy flight commander, branch chief, and project manager of several advanced air and space system technology development efforts.
(E-mail address: email@example.com)
Mr. Josef Oehmen is a research scientist with MIT’s Lean Advancement Initiative. His focus is risk management in the value chain, with a special interest on lean product development and program management. He holds a PhD from the Swiss Institute of Technology (ETH Zurich), a master’s degree in mechanical engineering from the Technical University of Munich, as well as an MBA from the Collège des Ingénieurs in Paris.
(E-mail address: firstname.lastname@example.org)
Mr. Ricardo Valerdi is an Associate Professor of Systems and Industrial Engineering at the University of Arizona. His research focuses on systems engineering metrics, cost estimation, test and evaluation, human systems integration, enterprise transformation, and performance measurement. He received a PhD in Industrial and Systems Engineering from the University of Southern California. Dr. Valerdi is the co-Editor-in-Chief of the Journal of Enterprise Transformation and has served on the Board of Directors of the International Council on Systems Engineering.
(E-mail address: email@example.com)
Aerospace Industries Association. (2009). The unseen cost: Industrial base consequences of defense strategy choices. Retrieved from http://www.aia-aerospace.org/assets/report_ industrial_base_consequences.pdf
Brier, G. W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78(1), 1–3.
Butts, G., & Linton, K. (2009, April). NASA’s joint confidence level paradox: A history of denial. Paper presented at the NASA 2009 Cost Estimating Symposium, Kennedy Space Center, Orlando, FL. Retrieved from http://science.ksc.nasa.gov/shuttle/nexgen/ Nexgen_Downloads/Butts_NASA%27s_Joint_Cost-Schedule_Paradox_-_A_History_of_Denial.pdf
Cancian, M. (1995, Summer). Acquisition reform: It’s not as easy as it seems. Acquisition Review Quarterly, 2(3), 189–198.
Carter, A. B. (2009). Directive-type memorandum (DTM) 09-027 – Implementation of the Weapon Systems Acquisition Reform Act of 2009 [Memorandum]. Retrieved from http://www.dau.mil/homepage%20documents/USA006945-09_signed.pdf
Carter, A. B. (2010). Better buying power: Guidance for obtaining greater efficiency and productivity in defense spending [Memorandum]. Retrieved from http://www.acq.osd.mil/docs/USD_ATL_Guidance_Memo_September_14_2010_FINAL. PDF?transcriptid=4648
Cummins, J. M. (1977). Incentive contracting for national defense: A problem of optimal risk sharing. The Bell Journal of Economics, 8(1), 168–185.
Department of Defense. (2008). Operation of the defense acquisition system. Department of Defense Instruction 5000.02. Retrieved from http://www.dtic.mil/whs/directives/ corres/pdf/500002p.pdf
Department of Defense. (2011). Fiscal year 2012 budget request: Overview. Retrieved from http://comptroller. defense.gov/defbudget/fy2012/FY2012_Budget_Request_Overview_Book.pdf
Department of the Air Force. (2007). U.S. Air Force cost risk and uncertainty analysis handbook. Retrieved from https://acc.dau.mil/adl/en-US/316093/file/46243/AF_Cost_Risk_and_Uncertainty_ Handbook_Jul07pdf
Fox, J. R. (1974). Arming America: How the U.S. buys weapons. Cambridge, MA: Harvard University Press.
Garvey, P. R. (2000). Probability methods for cost uncertainty analysis: A systems engineering perspective. New York, NY: Marcel Dekker.
Gates, R. M. (2011, January). Statement on department budget and efficiencies. Speech delivered at the Pentagon, Arlington, VA. Retrieved from http://www.defense.gov/speeches/ speech.aspx?speechid=1527
Government Accountability Office. (2008). Defense acquisitions: A knowledge-based funding approach could improve major weapon system program outcomes (GAO Report No. 08-619). Retrieved from http://www.gao.gov/new.items/d08619.pdf
Government Accountability Office. (2009). Defense acquisitions: Charting a course for lasting reform (GAO Report No. 09-663T). Retrieved from http://www.gao.gov/new.items/d09663t.pdf
Hubbard, D. W. (2010). How to measure anything: Finding the value of “intangibles” in business (2nd ed.). Hoboken, NJ: John Wiley & Sons.
Kahneman, D., & Tversky, A. (1984). Choices, values, and frames. American Psychologist, 39(4), 341–350.
McAfee, R. P., & McMillan, J. (1986). Bidding for contracts: A principal-agent analysis. The RAND Journal of Economics, 17(3), 326–338.
Russo, J. E., & Schoemaker, P. J. H. (1992). Managing overconfidence. Sloan Management Review, 33(2), 7–17.
Samuelson, W. (1986). Bidding for contracts. Management Science, 32(12), 1533–1550.
Scherer, F. M. (1964). The theory of contractual incentives for cost reduction. The Quarterly Journal of Economics, 78(2), 257–280.
Valerdi, R., & Blackburn, C. (2009, July). The human element of decision making in systems engineers: A focus on optimism. Presentation at the 19th Annual International Council on Systems Engineering Symposium, Singapore.
Weitzman, M. L. (1980). Efficient incentive contracts. The Quarterly Journal of Economics, 94(4), 719–730.
Williamson, O. E. (1967). The economics of defense contracting: Incentives and performance. In R. N. McKean (Ed.), Issues in Defense Economics (pp. 217–256). National Bureau of Economic Research. Retrieved from http://www.nber.org/chapters/c5165.pdf
- For practical purposes, the expected profit calculations were cut off at the 99 percent confidence levels because the 100 percent confidence levels theoretically extend to infinity.
- The Nobel Prize is not awarded posthumously; otherwise, it is generally regarded as a given that Tversky would have shared the honor.