To print a PDF copy of this article, here.
By Ivan Bembers, Ed Knox, Michelle Jones, and Jeff Traczyk
The first in a series of articles
It seems like every year the writing of budgets sparks proposals to eliminate the Earned Value Management System (EVMS) requirements on federal acquisitions as a way to save millions of dollars. People suggest the government can do away with EVMS in favor of more efficient and affordable management techniques. But is there a basis for these assertions? For more than 50 years, the Department of Defense (DoD) has recognized the power of both EVMS and the Cost/Schedule Control Systems Criteria (C/SCSC), the forerunner to EVMS, and has kept EVMS requirements in place to promote sound planning and effective program execution.The Government Accountability Office (GAO) has consistently cited EVMS as providing a comprehensive early warning of potential cost and schedule overruns. Furthermore, most major aerospace and defense (A&D) industry partners have made the business decision to adopt EVMS as a standard way of doing business on all types of development work, including commercial, fixed-price, and government cost type contracts. Why then do we hear the yearly grumbling about the burden of EVMS on government programs?
In 1994, Coopers & Lybrand/TASC (CLT) performed a study that included an assessment of the cost of C/SCSC. The CLT study, often cited as the definitive source in this matter, concluded that there was a 0.9 percent DoD regulatory cost premium for C/SCSC on government contracts. (The study noted, however, that the majority of the cost premium resulted from excessive requirements that were not inherent in C/SCSC.) Following that effort, Dr. David S. Christensen consolidated a number of other studies in The Costs and Benefits of the Earned Value Management Process and identified the cost of EVMS to be somewhere between 0.1 percent to 5 percent of the contract value. While these studies provide excellent information, all of their supporting data was gathered before 1996, when industry took more ownership of EVMS. In December of that year, Under Secretary of Defense (Acquisition and Technology) Dr. Paul Kaminski accepted industry’s 32 guidelines for EVMS and rolled them into the 1997 DoD Instruction 5000.2R. By July 1998, the guidelines were formally issued as an American National Standards Institute/Electronic Industries Alliance document, creating a national EVMS standard that was applicable beyond DoD programs. To this point, in 2014, the National Reconnaissance Office (NRO) Earned Value Management Center of Excellence (ECE) examined a large number of its major acquisitions and discovered that every prime contractor reviewed had an internal EVMS threshold for in-house, commercial or fixed-price efforts that was much lower than the requirement for NRO acquisitions (Figure 1).
If major industry partners within A&D rely on and use EVMS for government cost type, fixed-price and commercial efforts, and they have a management control system in place, then several questions need to be answered regarding any real or perceived additional costs of implementing EVMS on government programs:
- What are the differences in applying EVMS on a government cost type contract versus a commercial or fixed-price contract (including reporting requirements)?
- What are the underlying costs of these differences?
- What is the government value and derived benefit of these differences (i.e., additional deliverables or requirements that substantially help manage the program)?
- Are there opportunities to generate better efficiencies for these differences, especially in areas with claims of high cost and low value?
Better EVMS Implementation Study to address these questions. The JSCC was established by the Under Secretary of Defense for Acquisition, Technology, and Logistics and the Under Secretary of the Air Force as a joint government and industry forum with a commitment to affordable, accurate and credible cost estimating on space systems. The JSCC also actively addresses cost estimating and earned value management issues with the goal of improving cost-estimating accuracy and the betterment of earned value management practices, which in turn affect budget realism and improve schedule and program execution.
Due to the study’s extensive scope, the JSCC divided it into two parts (Figure 2). Phase I (completed in 2015) targeted industry and was designed to understand the delta implementation cost impacts of EVMS required for a government cost type contract versus EVMS performed on a commercial, internal, or fixed-price effort. Phase II (completed in 2016) targeted the federal government program managers (PMs) and focused on understanding how those PMs value and use EVMS products and management activities (P&MA).
During Phase I, the JSCC collected information from 46 separate space programs ranging in value from $20 million to more than $5 billion at Ball Aerospace, Boeing, Northrop Grumman, Lockheed Martin and Raytheon. This phase used survey responses to analyze 78 specific cost areas identified by industry as the key cost drivers (real or perceived) for applying EVMS on government cost type contracts. As shown in Figure 3, nearly 73 percent of survey responses identified the cost areas as No or Low Impact on the cost of EVMS, and the data identified government program management as the primary stakeholder driving High and Medium Impacts. Furthermore, cost impacts were scattered among all 78 cost areas, and no single cost area was identified as a High or Medium Impact across a majority of the programs that participated in the study.
Using Phase I data, the JSCC identified three overall themes regarding the cost of EVMS. First, Control Account (CA) level (size and number) significantly affects the cost of EVMS. Second, program volatility and lack of clarity about the program’s scope as well as funding uncertainty may affect the cost of EVMS, just as any other program management discipline. Third, volume of reviews (including surveillance, compliance, and Integrated Baseline Reviews [IBRs]) as well as the inconsistent interpretation of the 32 Guidelines affects the cost of EVMS (this theme could have two separate parts, but was originally based on industry’s interpretation of all government reviews). In April 2015, these themes were published in Better EVMS Implementation Themes and Recommendations along with specific recommendations to reduce cost.
Government and industry EVMS subject-matter experts (SMEs) also made several other observations regarding Phase I:
- Inconsistent government PM application of EVMS requirements appears to be the leading driver of High and Medium Impacts to the cost of EVMS.
- Inconsistent assessment of the materiality of Surveillance Review findings can affect the cost of EVMS (e.g., reviewer’s experience level, approach, etc.).
- The JSCC survey data does not substantiate the numerous anecdotal perceptions of major earned value-related cost impacts (e.g., IBRs cost too much, etc.).
During Phase II, the JSCC interviewed 32 government PMs at Air Force Space and Missile Systems Center, the National Aeronautics and Space Administration, and the NRO to assess the government value of EVMS. The JSCC assessed 12 specific EVMS P&MA ranging from Earned Value Data by Work Breakdown Structure (WBS), commonly known as Contract Performance Report/Integrated Program Management Report (CPR/IPMR) Format 1 to Over Target Baseline/Schedule (OTB/OTS). As shown in Figure 4, the results indicated that Integrated Master Schedule (IMS), the IBR and EVM Metrics were the most highly valued P&MA by government PMs. Even the lowest scoring P&MA, such as Earned Value Data by OBS (CPR/IPMR Format 2) and Integrated Master Plan (IMP), were identified as having medium value.
In September 2016, the JSCC updated Better EVMS Implementation Themes and Recommendations to include a series of recommendations on ways to increase value for each EVMS P&MA assessed in Phase II. These recommendations are rooted in the use of best management practices as well as education on how to use these EVMS P&MA better. As with Phase I, government and industry participants also provided additional observations:
- Most government PMs have a strong appreciation of EVMS—in many cases, they assess P&MA at the highest possible level and identify a heavy reliance of EVMS metrics during program execution.
- Most government PMs recognize the value of the IBR to generate a valid and executable Performance Measurement Baseline (PMB).
- Some government PMs do not fully recognize how surveillance can support their need to provide higher quality data (e.g., one PM contends that since he is continually walking the factory floor, he does not learn anything new from an independent surveillance).
Once Phase II was completed, the JSCC relied on government and industry SMEs to integrate the analysis of data collected during both phases of the study. Using a matrix of the Phase I cost areas versus the Phase II EVMS P&MA, the SMEs applied the premise, “The Customer requirement for EVMS Product/Management Practice X can influence Cost Area Y,” to determine the direct relationship of a particular EVMS P&MA with a specific cost impact. While the assessments were subjective, the SMEs required consensus on 936 specific matrix intersections and discussed all dissenting opinions to generate the best possible evaluation for each intersection.
As shown in Figure 5, the result of this synthesis shows that each EVMS P&MA is in the High-Value and Low-Cost quadrant. Additionally, analysis of Phase I and Phase II data also indicates that every EVMS P&MA except Surveillance, IBR, and Data by WBS shares 100 percent of its associated cost impacts with other P&MA (Figure 6). This means that even though a particular product such as Data by OBS may have a lower government value, the elimination of this product will most likely have a limited effect on reducing the cost of EVMS since its associated cost impacts will still exist on other P&MA.
Although the JSCC performed its study solely on space-related programs, the contractors who participated in Phase I are the same industry partners who build and deliver systems across the federal government. Likewise, many of the PMs interviewed had experience working in other commodity domains outside of space, representing civil agency and/or DoD environments. Therefore, the JSCC study results arguably apply to any acquisition domain.
So what does this mean in terms of the cost of EVMS? Since contractors could only provide level of cost impact instead of specific dollar values in the JSCC study, it is difficult to give an exact answer. However, if CLT is considered the “Gold Standard” regarding the cost of EVMS, the JSCC study is fairly definitive as to why the cost of EVMS on cost type contracts should be significantly less than the 0.9 percent identified in 1994.
First, during the 1990s, CLT based its results on a DoD-mandated C/SCS, while JSCC assumes that industry owns EVMS and considers it to be a best practice on all types of efforts (not just government cost-type contracts). Second, CLT incorporated all aspects of the costs of establishing, maintaining and using C/SCSC, while the JSCC study was established to identify the “delta” cost impact for EVMS on government versus other contract efforts (assuming the contractor already has an EVMS management system in place). This means that the cost impacts identified by the JSCC represent only a portion of those identified by CLT. While the JSCC recognizes that there is an expense associated with designing and implementing an EVMS management system, it is considered a one-time nonrecurring expense that should not be a liability to the government since a company should have some type of management control system in place to operate. In his 2010 publication, Earned Value Management: A Global and Cross Industry Perspective on Current EVM Practice, Dr. Lingguang Song stated that 69 percent of his 420 studied groups voluntarily used EVMS. This leads us to think that a growing A&D company will implement EVMS not only to support future government work, but because it is the most prudent thing to do for a self-organizing, competitive and profit-driven enterprise.
The bottom line is that there is no smoking gun to show that removing EVMS requirements from a government cost type contract will result in substantial cost savings. While the JSCC study does show a few higher EVMS-related cost impacts for a handful of programs, it does not identify any systemic High or Medium Cost impacts that affect a majority of the programs that participated in the survey. In almost every case where higher cost impacts do exist, they are typically driven by specific contract requirements.
When there is suspicion that high EVMS implementation costs exist on a program, several questions should be asked before drawing any conclusions:
Does the contractor use EVMS to manage? If so, is the contractor using its management system to support commercial, fixed-price, or internal efforts? Why will it be more expensive on a cost type contract? Ask for specific details. Identify the key cost drivers and obtain a basis of estimate.
If surveillance is identified as a key cost impact, what is the driver? Before discussing the cost of surveillance, what is a reasonable and appropriate level of corporate investment versus direct program cost in the maintenance of EVMS? Do industry and the government have the same expectations for EVMS and consistent interpretation of the guidelines? Periodic independent surveillance with timely resolution of issues is part of implementing a reliable and healthy management system.
If the IBR is identified as a contributing cost impact, what is the driver? How many people will be involved to support the review? How will their time be spent differently from normal day-to-day program management execution? Developing a program baseline is critical regardless of whether or not an IBR is scheduled.
If there is a request to eliminate the EVMS requirement or some aspect of EVM that is already on contract, what credit will be given back to the government? If a specific requirement, management activity or report is eliminated, will there be an actual reduction of cost and personnel on the program? Are those individuals identifiable by name? If something is removed from a contract, the government should expect to pay less.
Have both government and industry PMs reviewed ways to decrease the cost impact and/or increase the value of EVMS? Is the size and number of CAs optimized for risk and span of control? Are PMs aware of additional costs created by unique reporting requirements? Have PMs read and reviewed the results and recommendations of the JSCC study (available at www.acq.osd.mil/evm/resources/Initiatives.shtml)? This study provides practical recommendations and stakeholder actions to reduce costs and improve value, and may be helpful in identifying additional EVMS efficiencies.
Admittedly, using earned value to manage a program is not as glamorous as flying jets or working launch operations. However, the JSCC Better EVMS Implementation Study offers objective evidence that when EVMS is properly maintained and the data is optimally used, EVMS provides a high-value and low-cost management practice that (1) supports the delivery of valuable systems to the warfighter and (2) helps protect the American taxpayer from wasteful spending.
Bembers is the chief of the National Reconnaissance Office (NRO) Earned Value Management (EVM) Center of Excellence (ECE). He is responsible for EVM system acceptance and surveillance reviews, facilitating integrated baseline reviews and supporting programs’ use of EVM across the enterprise. Knox, Jones and Traczyk support EVM at the NRO.
To print a PDF copy of this article, here.