To print a PDF copy of this article, click here.
The Center for Army Acquisition Lessons Learned (CAALL) was established within the U.S. Army Materiel Systems Analysis Activity (AMSAA) to collect, analyze and disseminate acquisition lessons learned. The center serves as the authoritative source for timely, real-world acquisition lessons learned to enhance the performance of the Army’s project offices in support of the warfighter.
The Center for Army Acquisition Lessons Learned … provide[s] a record of acquisition experiences in order to allow others within the acquisition community to understand what occurred and avoid previous mistakes, as well as to provide the basis for making improvements.
The mission originated with the 2010 Army Acquisition Review, which repeatedly cited the need for a centralized source for lessons learned across the acquisition life cycle. The report stated that there are lessons learned within the Office of the Assistant Secretary of the Army for Acquisition, Logistics and Technology (ASA[ALT]), the test community and other acquisition stakeholders, but they are all dispersed and need to be synthesized. In addition, the report stated there is no formal way to track successes, analyze failures and develop best practices from historical programs. Therefore, the acquisition community needs a robust, readily accessible database and associated analytical capability to extract relevant information. The report recommended that a Center for Army Acquisition Lessons Learned be established to provide a record of acquisition experiences in order to allow others within the acquisition community to understand what occurred and avoid previous mistakes, as well as to provide the basis for making improvements.
Consequently, the Army Acquisition Executive (AAE) distributed a memorandum in January 2012 directing all Army acquisition programs, regardless of Acquisition Category (ACAT), to conduct After Action Reviews (AARs) and document lessons learned following all milestone reviews and program terminations. The memorandum also directed AMSAA to establish CAALL to collect these lessons via a Web-enabled database and to conduct analysis on the lessons learned.
Acquisition Lessons Learned Portal (ALLP) and Lessons Learned Collection
CAALL has established the ALLP as the authoritative source for Army acquisition lessons learned. The ALLP aims to serve as a knowledge management tool for the program executive offices (PEOs) and their project offices, as well as the broader acquisition community. The primary function of the portal is to allow easy input and retrieval of lessons learned. To facilitate collection of acquisition lessons learned, the ALLP provides an online form for lesson submission, as well as a downloadable form that can be completed offline and then uploaded to the portal. This allows the form to be filled out and distributed through different offices for review prior to submission if needed.
The main fields on the form include the Lesson Learned, Background, Recommendation, and Cost, Schedule and Performance Impacts. The Lesson Learned field is a concise (maximum of 200 characters), specific and actionable statement that describes the knowledge the author gained through the experience that can benefit other programs if shared and reapplied. This allows readers to easily and quickly identify the lesson and determine whether they would like to read further in the Background and Recommendation fields. The Background for the lesson describes the events observed or the actions taken and why they were taken. The Recommendation field provides details on how the lesson can be reapplied in the future and how it can benefit other programs or organizations. The form also has fields for capturing impacts to the program’s cost, schedule and performance. This information allows users to get an idea of the possible impact of the recommendation on another program if reapplied and allows CAALL analysts to identify those issues having the largest impacts on programs. In addition, the form collects metadata for the lessons, such as phases and milestones of the acquisition life cycle, categories, key words and ACAT level to which the lessons apply. This information aids users in finding lessons that may be applicable to their programs or types of work. It also is critical to CAALL’s analysis processes, as it allows the team to look at frequently used categories and key words to help identify trends within the lesson-learned submissions.
The ALLP includes a lessons-learned search page, where users may easily find lessons pertaining to their interests through a text-based keyword search. Users may refine their search criteria using filters for the category, ACAT, milestone and phase to which the lesson applies. The search will return a table of lessons along with any other information/fields the user specifies should be included in the search results table. Users may click on the Lesson Learned text for each lesson that appears in the search results to view the full lesson-learned record.
The portal includes other collaborative tools, such as a document repository and user forums. In the document repository, users may share detailed documents pertaining to their lessons learned, such as lessons-learned reports, useful templates, guidance, etc. In the user forums, users may post questions and informally discuss acquisition issues with other members of the acquisition workforce.
The ALLP includes two Spotlight Zones—Web pages focused on a particular hot topic in acquisition. The Spotlight Zones aim to provide the acquisition community with specific information that will aid programs in those areas of acquisition that are receiving significant attention in the acquisition world. The current Spotlight Zones include Reliability and Modeling and Simulation and provide lessons learned, case studies, links, guidance and tools pertaining to these two topics.
The Spotlight Zones aim to provide the acquisition community with specific information that will aid programs in these areas of acquisition that are receiving significant attention in the acquisition world.
Populating the Portal with Lessons
Upon establishment of the ALLP, CAALL began to populate the acquisition lessons-learned database with historical lessons from existing sources. The team gleaned lessons from various reports, such as the RAND Corp.’s report on lessons from the Future Combat Systems as well as Government Accountability Office reports. CAALL has had continuous interactions with the PEOs, as well as the greater acquisition community, to solicit lessons learned based on their real-world experiences. CAALL regularly attends Army Systems Acquisition Review Council meetings to stay informed on programmatic decisions and gather lessons or potential topics for lessons to be developed by the project manager (PM). In addition, PEOs and project offices have begun to populate the database with lessons from AARs following milestone reviews as directed by the ASA(ALT) memorandum, as well as informal, unprompted lesson submissions. The ALLP currently has over 600 users and provides access to more than 500 lessons learned.
Analysis and Dissemination
A key element of the acquisition lessons-learned mission is the analysis of lesson-learned submissions. This includes trend analysis of lesson submissions and deep-dive analyses of specific topics or trends, as well as case studies of particular Army acquisition programs. The AAE has recognized the need to continually identify the top five issues affecting Army programs and the need to have data to support these findings. CAALL synthesizes lessons and identifies trends to provide to ASA(ALT) in an effort to address systemic challenges and provide the basis for acquisition policy changes and strategic decisions. CAALL has begun to conduct deep-dive analyses of particular acquisition issues that have emerged from the synthesized trends. One such trend that has been identified is that documentation preparation and approval processes are resource drains for project offices. Consequently, CAALL is conducting a deep-dive study on acquisition documentation requirements and staffing in an effort to determine which documents cause the most issues, where duplication exists in documentation requirements, and where there are inefficiencies within the staffing and approval processes. Analyses such as this will be presented to ASA(ALT) leadership to provide detailed findings and potential recommendations for process changes. Furthermore, CAALL conducts case studies on programs that have had significant learning experiences that led to increased attention from Army leadership. Thus far, case studies have been completed on the Long Endurance Multi-intelligence Vehicle (LEMV) and the Armored Multi-Purpose Vehicle (AMPV). Findings from these efforts are available through the ALLP.
In addition to the ALLP’s lesson-learned search page, lessons are disseminated through periodic bulletins and article publications. CAALL distributes new lessons and other information on the current activities of the team through a quarterly Acquisition Lessons Learned Bulletin, which is provided to all ALLP users and Acquisition Lessons Learned stakeholders. Current readers include the PEOs and other acquisition organizations, such as the Army Test and Evaluation Command, Training and Doctrine Command, and the Defense Acquisition University. CAALL also regularly provides, to the Army AL&T magazine, articles that tie lessons learned and best practices into the magazine’s current theme. In an effort to push lessons to the project offices, CAALL has prepared “Just In Time” lesson-learned packages, each of which contains a set of lessons grouped by category (such as contracting, test and evaluation, systems engineering, etc.) pertaining to a particular phase of the acquisition life cycle. These packages will be available through the ALLP, as well as disseminated to a point of contact at each PEO so that PMs and their staffs may be provided with relevant lessons learned up front when entering a new phase of the acquisition life cycle. These packages will facilitate the sharing of repeatable good practices and knowledge from past mistakes with other project offices so they may benefit from the knowledge of those programs that have completed that acquisition phase.
Current Acquisition Lessons Learned
The ALLP houses a wide range of acquisition lessons learned from across the acquisition life cycle. Lessons pertain to a variety of topics, such as program management, technology and engineering, contracting and financial management. Highlighted below are a few lessons currently available in the ALLP.
Oversight, Review and Documentation
One of the largest trends within the current collection of Army acquisition lessons learned is the need for early and efficient milestone and documentation preparation. Programs have repeatedly cited the need to coordinate with stakeholders early, utilize documentation Integrated Product Teams and tracking tools, and know what is required for the milestone review. One program reported that the PM often needs to allocate critical resources to produce and staff, or obtain waivers, for documents that are not relevant to the program. The program had to meet traditional documentation requirements for its milestone review, which could have been averted if Army leadership would allow certain requirements (such as the Corrosion Prevention and Control Plan, Program Protection Plan, and Clinger-Cohen Compliance) to be declared inapplicable for programs that have existing materiel solutions and/or that don’t have certain components (such as electronics). Prior to the milestone review, the program had a materiel solution that was currently in use in the field, and contracts were already in place and producing the system to support deploying warfighters at a rate equivalent to Full Rate Production. However, the program still had to complete or obtain waivers for statutory, regulatory and policy requirements to achieve the milestone. The program recommended that PMs seek ASA(ALT)/Headquarters, Department of the Army, approval for the Milestone Decision Authority to declare certain documents to be inapplicable for certain programs instead of requiring a waiver or streamlined version of the document. A waiver or streamlined version can take almost as much time to prepare and staff as the traditional document. This would have saved the program many hours that could be used in other initiatives and would have allowed the milestone to be executed months earlier.
Another Army program reported that PMs need to be proactive and deliberate in initiating and establishing an Earned Value Management System (EVMS) for production activities at arsenals and depots. Proper Earned Value Management metrics were not established at the beginning of the program between the PM and the arsenal. The product office did not have EVMS or a similar management tool in place to establish a planning baseline or to measure cost and schedule performance over time. The lack of these tools contributed to what became a more than $41 million cost overrun and a 10-month schedule slip. The arsenal relies on a logistics system that proved inadequate for tracking earned value, defining the estimate at completion and managing end-to-end parts acquisition, production and costs. The system would not allow for material to be charged to a program until it is used, which can be several months after it has been purchased. Since the system has no automatic triggers or warnings, the arsenal would be able to continue to charge against a Military Interdepartmental Purchase Request (MIPR) even if funds had been exhausted. Consequently, the program provided the following recommendations for applying Earned Value Management type oversight:
- Conduct Start of Work Meetings and incorporate EVMS up front.
- Execute disciplined routine Program Management Reviews that focus on cost, schedule and performance.
- Utilize Integrated Master Schedule/Integrated Master Plan tools to measure program performance.
- Update cost estimates as the program evolves and every time the scope changes.
- Establish a measurable baseline at the outset of the program.
- Provide monthly cost reports to Life Cycle Management Command, customers and installation Commanders.
- Continuously refine metrics and reporting.
Test and Evaluation
Another trend that has begun emerging from acquisition lessons submitted by the PEO community is the need to consider test efficiencies during test-plan development. This includes increasing the number of test articles, combining different types of tests, using test data from similar programs and using modeling and simulation. One program reported that using similar program-and-design-level test data can reduce the number of required tests for the fielding of a system. The program had requirements to undergo some very expensive tests of multiple systems and subsystems. The program initiated a study to evaluate alternative solutions that would satisfy the testing requirements in the most cost-effective way. For this study, the program reviewed all test data from similar systems that had undergone these types of tests so the program could predict through test knowledge the results of the test. These predictions and results by similarity were briefed to the testing board, and a reduced test matrix was proposed leveraging these similar program and past design-level test results. The final test matrix required only 33 percent of tests originally planned/required, and the tests that were ultimately required were easy to fit into the schedule for fielding the system. The program recommended that other programs review their design tests and similar system tests to leverage this test data when putting together the required test matrix to meet materiel release requirements. The benefits of understanding the testing and ways to predict results based on similarity or design-level tests saved the program $1.86 million in hardware and test-range costs. In addition, the program benefited from a shortened schedule and reduced travel costs to meet the schedule required for materiel release of the system.
The ALLP continues to rely on valuable submissions from across the Army acquisition enterprise. These and other acquisition lessons learned may be accessed through the portal at https://allp.amsaa.army.mil/. The ALLP is open to all DoD military and civilians, and AMSAA welcomes you to contribute to this valuable mission by sharing your knowledge and experiences, as well as leveraging those lessons currently in the portal.