050613-article-10-lead

Why the TEMP Answers ‘Why?’


To print a PDF copy of this article, click here.

Author: Thomas L. Conroy II

The TEMP, or Test and Evaluation Master Plan, is the framework for an acquisition program’s Test and Evaluation (T&E) program. The TEMP is a four-part critical program document that links directly to the Acquisition Strategy and the System Performance Specification. The TEMP shows how the program will verify and validate the system requirements, whereas the Acquisition Strategy speaks to the management of the acquisition of the requirements, and the System Performance Specification guides the development of those requirements. The TEMP has a crucial role in ensuring that the system meets the users’ requirements and capabilities.

Each of the TEMP’s four parts is integral to answering the “why” questions surrounding the programming and planning for the developmental test (DT) and operational test (OT) and evaluation methods and resources. If the TEMP is written correctly, the order of the four parts also tells a story and answers these “why” questions effectively. If these “why” questions are used when creating a TEMP, it will be a very useful document for managing the test program.

Part I

The TEMP has four main parts. Part I of the TEMP is called the Introduction, but in reality it is everything one needs to know about the system being developed, tested, and evaluated. The relevant question answered by the information in Part I is “Why is this system needed?” One can see that Part I answers this question with the background information about the system and what capabilities and requirements are necessary to achieve its mission. Part I also uses this information to explain the rationale behind the prioritization of the capabilities and requirements for the system by explaining the nature of the threat and how the system combats it.

At the end of the developmental test program, we do not want to know that the system works well in a lab or controlled environment; we want to know what to expect when operating the system in the real environment.

Part II

Part II of the TEMP is known as the Test Program Management and Schedule. This section is very straightforward and it answers the primary “why” question of “Why does this testing need to be done now and under this budget?” This is important because it will constrain the amount of testing and evaluation that can be done on the program to prove that the system is effective and suitable in meeting its objectives. We’ll talk more about what it means to be effective and suitable in Part III. Part II sets the boundaries within which the test program needs to be accomplished successfully. This will be significant when trying to establish the best tradeoffs between how much testing is desired and how much testing is needed to evaluate what can be expected of the system’s true performance when used in the field.

Part III

The next section is Part III, the Test and Evaluation Strategy. This section is the heart and soul of the TEMP. But before we delve into this part, let’s talk a little bit about the two views of testing in terms of evaluated performance and the two views of testing in terms of evaluation focus.

When testing the performance of a system, the system is tested and evaluated for effectiveness and suitability. Effectiveness is the ability of the system to meet its mission and suitability is the ability of the system to be available to meet its mission. That is it in a nutshell. There are more detailed definitions, but those are the basics. An example would be that the effectiveness of a car is that it has the ability to get you to your destination within your timeframe, whereas the suitability of a car is that it is reliable and ready to drive and that it can be driven. If the car can get you to your destinations but you need to change the oil each trip, it may be effective but not very suitable.

Having said that, let’s discuss the focus of test and evaluation. The two main views of evaluation are from the points of developmental testing and operational testing. These views used to be very diverse, so much so that what is now Part III once was two separate sections, one for developmental testing and one for operational testing. Both views are now integrated into Part III.

050613-article-10-secondaryDevelopmental testing focuses on giving you what you asked for. It answers the question “Did I build it right?” Developmental testing is more to the point of meeting the requirement, or what was asked for, while trying to meet the needed capability. However, if the needed capability was not correctly translated into a specified requirement, then what was asked for may not meet that need. This second view, which answers the question “Did I build the right thing?” is called validation and is the focus of operational testing. It is easy to see how the two can diverge if the translated need is not fully resolved by the stated requirements. One example may be to state the need for a 200-square-foot room. If this is the only requirement, the requirement can be met, or pass verification and thereby developmental testing, by any combination of square footage in the room that totals 200. However a room that is 2 feet wide by 100 feet long may not suit your needs and would not meet validation or operational testing.

One can see how important it is that developmental testing and operational testing, or verification and validation, are given their due in supporting each other to gain the end user a system that meets all the requirements and capabilities to be both effective and suitable in the field. To this end, the operational test community focuses heavily on integrated test and evaluation. Integrated test and evaluation involve the integration of developmental testing with operational testing. This is accomplished in many ways, but one of the best ways is to make developmental tests look and feel like operational tests as much as possible. At the end of the developmental test program, we do not want to know that the system works well in a lab or controlled environment; we want to know what to expect when operating the system in the real environment. To do this, developmental testing environments need to be instituted to the greatest extent possible to simulate increasing levels of the operational environment, thereby decreasing the risk over the test program on the way to a fielding decision.

This is ultimately why there is a single combined developmental and operational test focus in Part III to reach both effectiveness and suitability. Verification must work with validation, and effectiveness must be balanced with suitability. Part III brings all these together to explain the test and evaluation strategy as a whole to include how many tests it will take, what methods of test and evaluation are necessary for each requirement and capability, and how the complete program balances to meet the need. Ultimately, Part III answers the “why” question of “Why is this combination of tests necessary to evaluate the system’s performance?”

Part IV

Part IV is the final part of the TEMP and it is called the Resource Summary. This is the point everything else was leading up to. This is what gets the plan done. Part IV is the description of the resources in terms of funding, test sites, and test assets that will be needed to meet the test and evaluation strategy described in Part III.
Part IV is the end of the document but also a beginning in terms of evaluating the TEMP to see if it is effective as a planning tool for the program once it has been written. If you ask “why” of each part, you should be able to find the answer in the previous part and be able to work your way back through the TEMP with all your questions answered. If you ask “Why are these resources in Part IV needed to accomplish this test program?,” you should be able to find all the answers in terms of what tests depend on those resources in Part III. If you ask “Why are these tests constrained the way they are in Part III?,” you should be able to find those answers in Part II. And if you ask “Why do these tests in Part III need to be conducted?,” you should be able to find those answers in Part I. Finally, if you ask “Why is the program constrained the way it is in Part II?,” you should be able to find those answers in Part I.

050613-article-10-secondary-2

Summary

The four-part TEMP is an effective tool in planning the test and evaluation program for a system in development. The TEMP has a crucial role in ensuring that the system meets the users’ requirements and capabilities that are documented in the System Performance Specification and acquired and managed through the Acquisition Strategy. It is a document that answers a number of questions about the nature of the test and evaluation program. In answering those questions while developing the TEMP, the TEMP becomes more effective as a management and planning tool supporting the entire system acquisition and management program. When it comes to the TEMP, it is OK to keep asking “why.”


To print a PDF copy of this article, click here.

Conroy is a professor of systems engineering (test and evaluation) in the Capital and Northeast Region of the Defense Acquisition University at Fort Belvoir, Va.

The author can be contacted at Tom.Conroy@dau.mil.

050613-article-10-ad

 

Comments

comments

Leave a Reply

Your email address will not be published. Required fields are marked *