To print a PDF copy of this article, click here.
For a long time now, the Defense Department has been using Technology Readiness Levels (TRLs) as a tool to assess the risk of including a new or advanced technology in one of our products. There is nothing wrong with TRLs except that they are only one input for a risk assessment and provide at best a crude indicator of the risk of using a technology in a product.
In many cases, TRLs tell us virtually nothing about whether we need to take additional action to reduce risk and what it will take to reduce a specific risk to an acceptable level. Let me give you three real-life examples I’ve seen over the last few years:
Example No. 1: An offeror on a missile program wants to incorporate a new infrared imaging array in a missile seeker. The technology will provide a significant performance enhancement. It employs a new material or perhaps just a larger array with a proven material. The offeror has produced several test arrays and incorporated them in laboratory test articles and in a prototype seeker that has been flown in a test article against a representative target. We would seem to have a technology that has reached the benchmark TRL 6; it has been tested in a prototype in a relevant end-to-end environment. What could be wrong? For a seeker material of this type, a critical question is its affordability as well as producibility, which usually is a function of the manufacturing processes’ yield percentage. Demonstrating that we can build a few test articles simply does not tell us enough about the viability of the technology for large-scale production and therefore about the wisdom of its inclusion in the design for an Engineering and Manufacturing Development (EMD) program.
Example No. 2: To support amphibious operations, a new ramp design is needed for a staging vessel that will be used to transfer ground combat vehicles from an amphibious ship to the staging vessel before they are loaded onto landing craft and deployed to shore. The intended ramp design is novel, but it does not include any new materials or design features that would expand the state of the art in any fundamental way. It is similar to other commercial and military designs but will be required to work in higher sea states than other similar structures.
Subscale models have been built and tested in tank tests, and extensive modeling and simulation work has been done to verify the design. This “technology” (or design) doesn’t meet the TRL 6 benchmark because it has not been tested in a relevant end-to-end environment. Should the program office be required to build a full-scale test article prior to entering EMD for the staging vessel? There is no way to know from the facts I have provided. Resolving this issue requires expert judgment about the degree to which the new design departs from proven capability, the risk of relying on model testing and simulation, as well as about the cost of designing, building and testing a pre-EMD prototype.
Example No. 3: New mathematical algorithms have been devised to fuse data from multiple onboard and off-board Intelligence, Surveillance and Reconnaissance (ISR) sources in a networked Command and Control (C2) system to be used on a new tactical strike platform. The success of these algorithms in substantially reducing the data processing loads on the C2 system will determine the viability of the design concept because of limitations on available power, cooling and volume on the aircraft. What must be accomplished prior to EMD to mitigate the risks of relying on these algorithms in the EMD design? If someone told you this technology was TRL 6, would that be enough to convince you that the risk was mitigated adequately? I hope not.
If someone told you this technology was TRL 6, would that be enough to convince you that the risk was adequately mitigated? I hope not.
One of the hardest and most important aspects of our jobs in developing and delivering new capabilities to the warfighter is risk management. A problem I’ve seen repeatedly is defaulting to a TRL assessment as a substitute for informed professional risk assessment and well thought-out mitigation plans, including specific knowledge points and decision criteria or exit/entrance criteria for the next phase of development. TRLs do not end the conversation about risk. TRLs may start the risk conversation, and they may provide a convenient shorthand benchmark, but they do not answer the question of whether the total risk of proceeding is acceptable, or define what work needs to be done to make the risk acceptable.
Some time ago I revised the technology assessment process that we require prior to major acquisition decisions, particularly the commitment to enter EMD, to place more responsibility on our Program Managers. I expect Program Managers to have a thorough and deep understanding of the technical risks associated with their programs and of the mitigation steps and resources required to reduce that risk. Technical risk considerations drive any number of program decisions, including: (1) the feasibility of requirements, (2) the need to conduct a Technology Demonstration (TD) phase, (3) the need for and value of competitive prototypes, (4) the specific accomplishments needed before entering EMD or initial production, and (5) the appropriate contract type. All this is Program Manager’s business, requiring judgment that goes well beyond any formulaic assessment of TRLs.
We also can’t assume that industry will take the needed steps to identify and reduce risk. A recent study of TD prototyping programs that I commissioned revealed that industry isn’t necessarily trying to reduce risk as its highest priority. When there is a competition, we can expect industry’s first priority is to win the competition. We have to make sure that winning the competition is synonymous with doing what the government needs done to identify risk and drive it down. The study showed that in many, in fact the majority, of the cases, industry was achieving an asserted TRL 6 benchmark for the government but not reducing the risk in the product that the vendor intended to build in EMD. This isn’t something we should blame industry for; we write the rules and we enforce them.
We will never have, and should not expect to have, risk-free programs. Our warfighters have the best equipment in the world because we take the risks inherent in doing things that have never been done before. Our technological superiority rests on this foundation. As acquisition professionals, we have to manage risk so we strike the right balance between stretching for new and better capabilities and limiting our goals to ones that are attainable and will be reached efficiently at acceptable cost. TRLs are just one of the tools we use to accomplish this task, and we should not rely on them for more than they can provide or think of them as a substitute for the professional judgments we have to make.