To print a PDF copy of this article, click here.
Delsing is retired U.S. Air Force lieutenant colonel and was the T-X (trainer jet) deputy program manager and helped to develop the application of the Multi-Attribute Decision Model techniques to the acquisition process for Air Force Materiel Command. The author expresses special thanks to Retired Lt Col Len Cabrera for the example used in this article, as well as the United States Air Force Academy Capstone team for its computer-based tool.
Under Secretary of Defense for Acquisition, Technology, and Logistics Frank Kendall and Secretary of the Air Force Deborah Lee James have introduced many new initiatives in an effort to improve U.S. Air Force Acquisition, including Better Buying Power, Owning the Baseline, Bending the Cost Curve, and others.
A common theme encountered in these initiatives, as mentioned by James in January 2015 is the “Cost Capability Analysis process.” The goal of this process is to use the knowledge of capability trade-offs to determine where a small trade in capability (e.g., top speed of an aircraft) could be adjusted for large cost savings. So how would a program manager (PM) go about doing this?
While the concept is fairly straightforward (just tell me where I can save some money without losing too much capability), the actual process to find these trade-offs can be somewhat daunting. How does a PM know where capability trade-offs can be made within a set of user’s requirements? Which trades provide the greatest value? How is an objective basis provided for requirement trade-offs?
One method of answering these questions is the Multi-Attribute Decision Model (MADM). The MADM uses an attribute hierarchy to assign a value score to each alternative. Alternatives then are compared based on the requirement attribute score and cost to determine which are most efficient (i.e., provide greatest performance for the cost, or lowest cost for given performance). Once these efficient alternatives are identified, it’s up to the PM, working with the stakeholders, to decide the proper trade-off between price and performance on the efficient alternatives.
The value score is computed using an attribute hierarchy. The first level breaks down the requirements by category, which is then weighted so that the percentages add up to 100 percent. Each requirement attribute within a category similarly is weighted within the category, again summing to 100 percent. Finally, specific performance metrics then are identified and assigned relative importance to achieving the requirement.
Put more simply, this method helps a PM and the stakeholders decide when more than one requirement drives the solution. What sounds complex actually is fairly straightforward. A car-buying analogy can demonstrate how it works.
Let’s say that you, as a PM, have been tasked with acquiring an automobile for your organization. The first step already has been done: Your requirements have been handed to you (Figure 1).
Figure 1. Example of Requirements for Acquiring an Automobile
Based on these inputs, it is time to develop the requirement attributes you’re going to evaluate. More specifically, you need to work with your stakeholders to develop those measurable attributes you will use to provide the capability requested. For example, both the Community
Outreach Representative and the Safety Office Advisory have requested a safe vehicle. If we use that input to create a “Safety” category, we can then look at some common safety features that the user may want in a modern automobile. In this case, we’ll use number of airbags and braking distance, given a speed of 60 miles per hour (mph).
Also note that some requirements may be eliminated at this stage for not really being as important as the stakeholders originally thought. In our example, it turns out storage capacity is not a high enough priority to call out specifically and therefore was eliminated early.
Once you have agreed to the requirement attributes, lead the stakeholder to prioritize the requirements. In this case, work out the relative value of each requirement category followed by each individual requirement attribute. Remember that the sum of the category weightings must equal 100 percent, and the sum of the attribute weightings within the categories must also equal 100 percent. (Figure 2).
Figure 2. Category and Attribute Weightings Within the Categories
Next, you need to determine where the stakeholders want the actual value of each of these requirements to be. It is helpful to “anchor” these curves with some questions. What value would be preferred (i.e., your target value)? What is the least acceptable value (i.e., any worse, and the requirement no longer provides any value)? What would be the ideal number (i.e., any better and no added value gained)? Using these reference points, where the worst value is assigned 0 points, and the best assigned 100 points, a “utility curve” can be created to describe the value space of that particular requirement.
In this case, the stakeholders told you that they would prefer no more than 19 days in the shop over the planned four-year ownership period. Ideally, they would like to keep it down to 12 days (in any shorter time, their favorite mechanic would suffer). However, should the mechanic need to be seen 26 or more days, the automobile would no longer provide the desired capability.
Based on this input, our utility curve looks like the Days in the Shop graph in the Reliability section of Figure 3. .While you have the stakeholders all together, you can work your way through each of the requirements to develop their utility curves. Some will be simple (e.g., is four-wheel drive [4WD] installed?), while some take more discussion (e.g., non-linear curve on airbags) (Figure 3).
Figure 3. Automobile Reliability, Safety and Functionality
Having completed the stakeholder inputs, you can now research the alternatives available. Using the best industry information available, other users and your own market research, you find four automobiles that may provide the solution you seek (Figure 4).
Figure 4. Comparisons of Reliability, Safety and Functionality
From here, it’s just a matter of crunching the numbers. Determine the number of points each attribute scores from the utility curves, and apply the attribute and objective weightings. When these weighted scores are added together, the result is a normalized score for each alternative.
For example, first find the global weighting for the “Number of Days in the Shop” attribute by multiplying the attribute weighting by the category weighting (40% x 40% = 16%). Next, look up the utility value of “Number of Days in the Shop” for Car A from the utility curve (86). Multiply this utility value by the global weighting for that attribute (86 x 16% = 13.76).
Doing the same for each attribute and adding those scores together will yield a normalized global utility score for Car A—in this case, 68. Complete the same calculations for each car. These normailzed scores of each alternative now can be plotted against cost to give the master cost-capability plot (Figure 5).
Figure 5. The Pareto Front in the Master Cost Utility Plot
What does this Master Plot tell us? First, Car A has the highest overall utility score. The theoretically ideal automobile would score 100 points (i.e., it meets or exceed the maximum utility scores in each category). In this case, Car A scored a total of 68 points, while Car B scored 65. From a purely requirements-based approach, Car A would be the best choice.
Second, Car C is the least expensive. Although lower in utility, it provides the most economical solution.
Third, Car B appears to provide some value at a midpoint in cost. If we assume that our ideal automobile (i.e., 100 points) also is ideally priced (e.g., $20,000), it would be located in the upper-left corner of the plot. By that reasoning, the closer we get to the upper-left corner, the better the solution. By drawing a line between those alternatives that score no lower in utility as we increase in price, we create the Pareto Front.
Any alternative that falls below this front—in this case, Car D—would be too expensive for too little capability.
This plot also is a good place to start a conversation with the user. In this case, we have three potential alternatives that provide good value for money based on the inputs provided. However, note that none of the three cars on the Pareto front has 4WD. Additionally, Car B falls above the requested maximum days in the shop. Car C provides no warranty value to the user with only a three-year warranty. This is where we can start the discussion of trading capability for cost. Note that this analysis is not sufficient as a basis for the ultimate decision. In the end, the PM must work through the trade-off discussions and use this method as one of many tools for the ultimate purchase decision.
Finally, the MADM technique provides a good tool to help define requirement value, normalize alternative performance, and start cost trade-off discussions. By having the relevant stakeholder score and weight requirements against each other at the start of requirement development, the PM can drive the user to hold early the difficult discussions on which requirements provide the greatest benefit and value. The analysis itself allows the PM to take those inputs to create a relatively objective discussion space where alternatives are scored based on cost and performance against predefined value. The Pareto Front technique provides the PM with the information key to an objective and value-based discussion on requirement trade-offs.
Additional analysis techniques can be performed, including individual requirement cost capability curves and sensitivity analyses. The U.S. Air Force Academy Operations Research Capstone class has developed a computer-based tool to help PMs complete the MADM analysis, and Air Force Materiel Command is developing a standardized process to facilitate the overall Cost Capability Analysis.
The author can be contacted at firstname.lastname@example.org.