loc-lead

The Effects of Incentives in Acquisition Competition on Program Outcomes

By William E. Novak and Harry L. Levinson

A review of acquisition program outcomes would make it appear that many acquisition programs are destined to experience recurring schedule slips and cost overruns, and produce poor quality systems. We decry these circumstances, but the acquisition community has had limited success in correcting them. However, the analysis of data collected from assessments of many software acquisition programs has produced insights into some of the most common recurring counter-productive program behaviors. One result of this research has been the identification of a set of misaligned incentives that are a significant force in driving acquisition programs toward poor performance.

These types of incentives affect all aspects of acquisition programs, including the competitive forces that are meant to reduce acquisition costs. The system for acquiring defense systems has many misaligned incentives that unintentionally promote program outcomes which are in direct conflict with the stated intent of the acquisition system. These occur in all phases of acquisition, from pre-award to sustainment.

Some scenarios illustrating the unintended consequences of acquisition
competition include:

  • The consolidation of multiple needs into single joint acquisition programs that contractors must underbid to win creates schedule pressure that drives cost and schedule overruns, and encourages stakeholder programs to opt out, undermining the value of the joint program.
  • Short-term performance incentives at the program management office can forestall sustainment planning, often omitting contractual delivery of key sustainment tooling—and as a result, may inadvertently lock in the development contractor as the only viable choice for sustainment.
  • This paper illustrates each of these dynamics with a narrative of the experience of an acquisition program, drawn from personal interviews with the program office and contractor staff.
  • The paper also describes ongoing research studying acquisition incentive issues using modeling and simulation techniques. Building on past work with qualitative models of adverse acquisition dynamics, we are building executable models of program behaviors that can be simulated, validated, and analyzed quantitatively. This research approach is described, along with plans to model candidate solution approaches to these dynamics, allowing the proposed solutions, including acquisition system policy changes, to be assessed for their efficacy in mitigating or resolving the problematic behavior before a candidate policy would be implemented.

Summary

Misaligned incentives influence many aspects of acquisition programs, including competitive forces, in ways that can undermine program goals. These include incentives for joint stakeholders to demand custom capabilities from joint programs and thus escalate cost and schedule, as well as incentives acting against effective sustainment planning that can lock in contractors for sustainment contracts. The paper analyzes these instances, and discusses ongoing work using modeling and simulation to analyze the effects of misaligned incentives, and ways of mitigating their effects.

The Effects of Incentives in Acquisition Competition on Program Outcomes

The common failure of acquisition programs to achieve their desired performance in delivering high-quality systems within cost and schedule constraints (“DoD acquisition outcomes,” 2005)—especially those developing software-reliant systems—is an all-too-common occurrence in modern government acquisition. These recurring failures have a direct adverse impact on the ability of the U.S. Department of Defense to be able to support the warfighter with the systems they need, and that the U.S. is capable of providing. Delayed systems withhold needed capability, and wasted resources drain budgets that could be used to develop other new systems.

While there is widespread recognition of the failures in acquisition programs, there is not general consensus as to the reasons for this. The Defense Acquisition Performance Assessment report (Kadish, 2006) acknowledges that the underlying causes for poor performance are still not well understood—and understanding them is the first step toward improvement.

The SEI is regularly asked to conduct Independent Technical Assessments (ITAs) to investigate the reasons why specific acquisition programs are seriously challenged, and even fail, despite the best efforts by government and industry to make them successful. These investigations have provided visibility into the processes and forces at work within these programs. Analysis by the SEI on data collected from over 100 ITAs of software-reliant acquisition programs has produced insights into some of the most common ways that programs encounter difficulties.
We know that acquisition programs do not fail solely for technical reasons. One set of significant reasons why acquisition programs may substantially exceed budget, overrun schedule, and ultimately fail may be due to organizational, management, and cultural issues (Madachy, 2008, Frangos, 1998). In the SEI’s direct engagements with large government acquisition programs we have seen pervasive recurring failure patterns due to systemic issues in the form of behavioral dynamics among individuals and stakeholder groups.

Acquisition Programs as Complex, Dynamic Systems

The focus of technical acquisition programs is most commonly on the complex systems being developed. In maintaining that focus, however, it may not be noticed that acquisition programs themselves are complex, dynamic systems. As such they can display unpredictable and even seemingly chaotic behavior. This results from the presence of feedback between the different elements of the organization, producing non-linear behavior that defies traditional mathematical analysis. The complexity of feedback, which is inherent in any system involving human beings who can spontaneously interact, coupled with time delays that obscure the relationships between cause and effect, can produce this unexpected behavior even in simple systems—which means that actions taken may have unintended side-effects that can worsen the problems they were meant to solve. What is important to realize, however, is that beneath that complexity there are recognizable recurring structures that drive these behaviors, which can be understood and manipulated. Past work done at the SEI in developing the “Acquisition Archetypes” (Novak & Levine, 2010; McNew, 2011)1  has described such recurring patterns of counter-productive organizational behavior based on observations made on actual acquisition programs.

Once we recognize that an individual acquisition program (and the entire acquisition system) is a complex, dynamic system, we can apply appropriate methods and tools to analyze and investigate the impact that forces such as competition might have if introduced. Through this analysis process we can investigate the unintended consequences that can result from the use of competition in acquisition. Some examples of consequences that affect government acquisition are:

Vendor “Lock-In”: Competition creates an incentive for companies to differentiate their products and systems from existing “standards,” often through the use of desirable vendor-specific extensions. Standards are inherently commodities, and thus for a vendor, standard adherence tends to drive down price, which becomes the primary differentiator. The incentive to produce custom extensions is to gain competitive advantage and price differentiation by using the extensions both to show superior capability, and also to create a “switching barrier” to “lock in” users by making the cost and effort to switch to a competing product high.

Underbidding: Underbidding on proposals is a common result of misaligned incentives among defense contractors that is continually reinforced. When contractors are able to win contract awards through underbidding and then generate additional revenue later in the program to recover the “lost” revenue from the underbid, this encourages other contractors to employ the practice to an even greater degree to remain competitive and win the next contract.

Structural Dynamics and Misaligned Incentives

In addressing these types of issues the Defense Acquisition Performance Assessment report (Kadish, 2006) states that in the defense acquisition system “Incentives are misaligned –PMs and contractors are not necessarily rewarded for decisions that lead to lower life cycle costs or provide a better balance between cost and performance.” The report continues on to claim that “Efforts to reform a system in an organization as large and complex as DoD must understand and address the root systemic causes of organizational and individual behaviors to be successful.”

Some of the work done at the SEI has concluded that the dynamic behaviors seen in many acquisition programs are based on two types of underlying and interlocking forces: misaligned incentives and structural dynamics.

Broadly speaking, misaligned incentives are aspects of the acquisition system such as its rules, regulations, policies, and guidelines that come into conflict when an acquisition decision-maker must trade-off maximizing the achievement of one objective against another. For example, misaligned incentives in acquisition programs can put individual or program-specific interests ahead of PEO or Service interests, turning planned cooperation into opposition, and producing poor acquisition outcomes.

Structural dynamics, while also an aspect of the acquisition system, are the natural, or even in some sense “physical” aspects, of developing and acquiring software-reliant systems. These structural aspects include such components as feedback loops and time delays. For example, diverting developers from new development tasks to fix defects that were injected as a result of working under heavy schedule pressure has the unintended, yet predictable feedback consequence of slowing development, and thus further increasing schedule pressure.

Many of the issues caused by misaligned incentives in acquisition are readily apparent. In one example, while the program office and the stakeholders both have incentivizes to minimize program schedule, in the preferred contracting vehicle for unprecedented development (i.e., Cost-Plus contracts) the contractor has an implicit incentive to maximize schedule in order to increase revenue and maintain a stable set of employees. In another example, when it comes to requirements, the program management office (PMO) wants to minimize requirements changes, while stakeholders (users) want to obtain all of the features they desire (regardless of whether they are in the original baseline), and the contractor (again in a Cost-Plus context) is happy to see the revenue benefits of both the additional new work, as well as the rework from requirements changes.

As an example of the combination of misaligned incentives and structural dynamics, we can think of the situation in which strong schedule pressure is exerted upon a software development manager by higher-level management and other stakeholders. The amount of that pressure that the manager then chooses to place upon the developers is a management decision that he or she makes based largely upon incentives that are often misaligned with the goals of the project: the potential effects of late system delivery on his or her career, the expected effects of the schedule pressure on the developer productivity and morale, and so on.

The “physics” of the underlying structural dynamic that contains both feedback and a time delay is: “If a developer is worked too hard, his or her productivity will drop off, and they may eventually leave.”  If the manager chooses to exert substantial schedule pressure on the developers, the schedule pressure will at first increase, but then ultimately degrade productivity as developers begin to “burn out” and possibly leave the organization. While the amount of pressure that is placed on the developers is a management choice that is the result of incentives, the results of chronic schedule pressure on the developers are not a choice—they are predictable and almost inevitable.

Taken together, misaligned incentives coupled with structural dynamics will drive decisions that seem (and may be) reasonable at an individual level, but may also have unintended consequences. The intent of increasing schedule pressure on developers may be to raise their productivity to meet a schedule—but the longer-term consequence is likely to be the opposite. The manager may not be aware of this consequence—or may make a calculated gamble that the personal consequences to himself or herself of trying to accelerate development through increased pressure on the developers will only become problematic after the next key deliverable is made. After that the schedule pressure will decrease, and thus the productivity benefits will be gained without suffering the worst of the potential costs of developer burn-out and turnover.

With both types of forces present in many different acquisition situations, their interdependence injects additional complexity with the potential for significant unintended results. The interactions of the two aspects frequently combine to produce the counter-intuitive and counter-productive outcomes that are often seen in acquisition.

Systems Thinking

One technique that is useful for analyzing complex systems with these characteristics is systems thinking, a method rooted in the system dynamics work pioneered by Dr. Jay W. Forrester at the Massachusetts Institute of Technology (Forrester, 1971). Systems thinking views systems as sets of components that have complex interrelations occurring between them—and these types of systems occur commonly in economics, the environment, business, politics, and organizations of all kinds. A key tool of systems thinking used in this paper is the causal loop diagram, which emphasizes the occurrence of feedback loops in systems. Causal loop diagrams illustrate the values and feedback loops present in a system, and how they interact with and affect one another, making them useful as an explanatory tool to present the sequence of events that occurs in a dynamic behavior. They are used here to illustrate the structure and flow of the dynamics in two example acquisition scenarios.

This paper explores ways to analyze the acquisition system, and methods to investigate the impact of competition on it. Using methods such as systems thinking, the presence of misaligned incentives and structural dynamics can be identified and analyzed for influences they have on the effectiveness of competition in acquisition. This paper will explore some pertinent examples of how this this modeling can be done and how competition, or the lack thereof, might play out in these scenarios.

Acquisition Competition Scenarios

There is no shortage of instances in which competition affects the performance and outcome of acquisition programs. These occur in all phases of acquisition, from pre-award to sustainment. Two useful scenarios illustrating the unintended consequences of acquisition competition are:

  • The consolidation of multiple needs into single joint acquisition programs that contractors must underbid to win creates schedule pressure that drives cost and schedule overruns, and encourages stakeholder programs to opt out, undermining the value of the joint program.
  • Short-term performance incentives at the PMO can forestall sustainment planning, often omitting contractual delivery of key sustainment tooling—and as a result, may inadvertently lock in the development contractor as the only viable choice for sustainment.

The following sections present more detailed discussions of these two scenarios, which have been adapted from actual acquisition programs. While no specific programs or individuals are identified so as to preserve confidentiality, the narratives presented are based on interviews conducted with program staff, contractors, sponsors, and others, and direct quotes are used when possible.

Scenario #1: “Consolidation into Joint Programs”

The first scenario presents a dynamic that involves the effect of the larger acquisition system context on the way acquisition programs are approached.

“Consolidation into Joint Programs” Scenario Narrative

The program was started in a context of a tightening defense budget, together with a growing need for improved interoperability as systems were becoming more interconnected. One response to these two trends was an increasing interest in the use of joint programs as a way to partially address both issues. Defense contractors responded to the shift by aggressively pursuing the joint program contracts, correctly recognizing that these larger efforts would be especially important in an environment that would feature fewer programs overall—and aggressive bids were an important strategy in winning the contract award.

Off and Running

The concept was an ambitious one, and had been under study for almost a decade before the Joint Program Office (JPO) was stood up. The new capability was originally conceived as a way of improving interoperability across a variety of platforms from the different Services by building a single capability that would be incorporated into each platform. This also solved the commonly understood problem that “…there wasn’t enough funding for the Services to each develop their own.” According to one of the Service stakeholders, “They decided that whatever they did would have to be done by all of the Services. They didn’t want the Services to all separately implement to one standard, as they had in the past—they wanted to develop the software once, and make sure that it would be done in the same way across all platforms and Services.” That way, the thinking went, they would “…pay for the capability only once, instead of many times” to avoid having “…50 disparate [capabilities] that the Services were developing,” with the problem being that “…you not only pay multiple times, but you get poor interoperability as well.” At the same time, the officials in charge “…may have been afraid that if they gave it to a Service, that they wouldn’t follow through with it” and produce something that would be generally usable.

Still, not everyone was fully on board, or completely sold on the joint concept. There were some serious implications of taking this approach to developing the new capability. It was clear to each of the participating programs that “…the [stakeholder program] becomes dependent on the group doing the [joint] capability—so they’re now dependent on [that] to achieve their own requirements for their platform.” For the Army, that meant they were now “…totally dependent on the [joint capability],” and “…if [it] fails, the rest of the house of cards comes tumbling down, and you’re back to Service capability.”

As a senior acquisition executive observed later, left to their own devices “The Navy would have gone with [their legacy system]” and “…the Army would have gone grudgingly…” along with the joint effort. It was common knowledge to those supporting the new program that “Without a joint requirement, it significantly increases the risk of a stovepipe approach” because “It’s easier to do a stovepipe system than to do joint.” As one senior official noted, “People don’t want joint programs—they want to control their own destiny.”

The Catch

One of the biggest challenges in getting a joint program off the ground is defining the scope of the new functionality that it will provide. While it would have been ideal to offer a capability that would not only be the union of the legacy system features that each of the platforms already enjoyed, but which would also add to that the new capabilities being developed, budgetary constraints indicated that would be difficult. For just such reasons, one of the joint program leads acknowledged that “Joint programs have always been focused on providing the lowest common denominator” across stakeholders in terms of functionality. This created a dilemma, because as another program lead commented, “The Services are insisting that the [new software] be capable of performing all of the functions that their legacy systems can do today…  They won’t change over to using the [new software] if it means that they will lose any capability.” Almost as soon as the CDD requirements had been defined, the joint program realized that “The Services want more in the system than what’s in the CDD.”

Knuckling Under

When the stakeholder programs began to come to the JPO to advocate for the specific requirements they needed that were outside the baseline the JPO had proposed, the JPO had a difficult choice to make: either agree to add multiple sets of custom requirements, and as a result slip schedule, drive up cost, risk, and complexity, and thus shake the stakeholders’ confidence in the JPO to the point where many stakeholder programs might decide to leave the joint program—or refuse to add the new requirements, and by doing so risk angering the stakeholder programs enough that they might leave the joint program.  Either option could lead to the failure of the joint program.

Initially the JPO was reluctantly willing to add the new requirements to the capability, preferring to avoid the immediate wrath of the stakeholder programs in favor of the longer-term cost and schedule growth issues. According to a senior official, “The Services meandered with the requirements process, and the JPO allowed it—they should have held the line.” It was clear that the stakeholder programs wouldn’t be satisfied until they each reached the same level of functionality that their legacy systems had provided, and that was going to require extending the requirements baseline. As the JPO noted, the joint program was “still chasing requirements from the Services that haven’t been formally required or requested.”

When a new colonel took over as PM at the JPO in the program’s fourth year, he resolved to run things differently, and as a result the joint program “…became very defensive about doing any additional work [that wasn’t already in the requirements], because doing it would delay them, and make them look bad.” Despite the fact that refusing to add still more [customized] functionality to the program was a discipline that most agreed was long overdue, “the Services didn’t like the Colonel holding the line on… functionality and funding.”

Unintended Consequences

With the additional requirements that had been added to the baseline, some recognized that the JPO now had “…a tough engineering problem, and no one appreciates the complexity, work, risks…” that it presented. They were even warned by a technical advisor that they were “…embarked on a very high risk software venture.” The consequences of that complexity and risk started to become apparent as development progressed, and one senior executive noted with growing concern that “…the schedule has been continually slipping over time. Every time they’ve gotten close to delivery, they’ve slipped.” The program cost was growing accordingly, following a pattern described by one Service stakeholder as “…double the cost, and multiply the schedule by a factor of 2-3, and [the program] fits right into that model. The schedule slips, and the cost grows.” This was alarming to the stakeholder programs who were “…worried about [the] cost, schedule and risk… [of] trying to deliver a combat system” when “a piece of this is being developed by an outside organization [i.e., the JPO] with a track record of sliding schedules and faulty systems, and this [capability] is going into the center of this combat system…” To put it simply, “The potential for [the capability] to harm [their platform] has the Navy scared.”

Defecting

As the schedule continued to slip and costs continued to rise, a key Army program stated that “Their confidence in the… [ability of the] JPO to deliver has eroded over the last year and a half” and that they had “very little confidence [the JPO] will be able to deliver what they say they will in the time they say they can.” The JPO noted the vagueness of the “lack of confidence” charge, with one acquisition executive suspecting deeper reasons for it, noting that “If… confidence were their issue, they could push for more/better testing—but then they might have to implement it.”

Given the situation, there was uneasiness on the part of several programs, as well as their Services. One program reviewer said “I’m not sure that the Navy really wants to do this,” and a Navy captain acknowledged that “We have to integrate it because OSD and JROC have said that we have to.” Still, the Army was more vocal about their concerns. They had made it clear to the JPO that “The Army wants to get the [joint] requirements out of their CDD. They want to have a way out.” With the amount of conflict that was starting to show at the review meetings, others began to wonder if the current approach could succeed. A member of a team that reviewed the joint program recommended that they should “Let the Army interface with their own system, and control their own destiny,” while one of the Service stakeholders believed that “If the Army goes off on its own, everyone loses. Title X gives the Services the right to be wrong, and they’ve certainly exercised that right.”

Unraveling

Not only were the total costs of the program increasing, but the prospect of losing one or more of the current stakeholder programs meant that the remaining programs would have to shoulder that additional share of the financial burden. As it was, according to one Service stakeholder, “we thought it would be $500M originally, and then it got blown up into a $1B program. That’s when the wheels started to come off, and the Service contributions started having to ratchet up.” The prospect of an additional cost increase, together with the delivery issues, was stretching the remaining stakeholders’ faith in the effort. As one program reviewer put it, “The JPO has had their share of delivery issues in the past, and the [stakeholder] programs paid the price for that in the past, and have lost confidence.” Another Service stakeholder was more blunt about the degree of conflict that was starting to emerge between the stakeholder programs and the JPO, saying “People get spitting mad, and are shaking with rage at some of the meetings.”

Collapsing

Ultimately, as a member of a program review team summed it up, “The key question is whether the Services want a joint solution… or do they want to go their own way and perpetuate their interoperability challenges?” One of the stakeholder program managers had admitted that the joint program was “…a train wreck in progress,” and stated, “I believe that it will fail.”  In looking at the level of hostility that had erupted between the stakeholder programs and the JPO, it was clear to one of the acquisition executives that “The Services don’t want this.” In fact, one of the Service stakeholders had recommended that the DoD should “Kill the program as we know it today, and salvage what they have.”

The words of that stakeholder proved to be prophetic when it was formally announced shortly thereafter that the program had been cancelled. Even beyond the failure of this specific program, however, was that despite the straightforward economic rationale for joint programs, there was a growing realization that what had happened was “symbolic of a deeper problem.” Joint programs were fundamentally more challenging than other types, and this program was just one example of the kinds of issues they would continue to see on other joint programs. This began to put the “needs consolidation” strategy into a different, and much less favorable light. As a senior acquisition executive who was responsible for the program, and who had made a strong effort to force the Services to work together, asked in the aftermath, “What else would you have had me do?”

“Department of Defense acquisition strategies that consolidate multiple capability needs into ‘single weapon system procurement’ force industry into ‘must win’ cost consolidations.”

“Consolidation into Joint Programs” Scenario Analysis

Joint programs have become common throughout the DoD as a way of both reducing costs and, in some cases, improving interoperability. According to former Secretary of Defense Robert Gates “…The military’s operations have become very joint – and impressively so…” (Bennett, 2009). However, those benefits have come at a price, and have given the concept of the joint program a poor reputation. As described by Kadish (Kadish, 2006), “Department of Defense acquisition strategies that consolidate multiple capability needs into ‘single weapon system procurement’ force industry into ‘must win’ cost competitions.” The concept of a “single weapon system procurement” that will address multiple different needs has come in many instances to imply a joint program. The underbidding that is frequently employed by the competing contractors and is then often captured in the contract award budget creates the initial schedule pressure that helps to undermine the joint program from the start. The inadequate funding can set the program up for significant cost and schedule overruns—thus creating opportunities for joint program participants to leave, blaming their departure on rising costs and slipping schedules, undermining the value of the program (by reducing the number of participants), and potentially causing its collapse.

Following the systems thinking approach of creating a causal loop diagram of the dynamic described by the narrative is a useful way of simplifying the behavior to its essential components. Figure 1 depicts the basic causal interactions and flows.

loc-2-figure-1To read the diagram as a story, ideally in a joint program as the Number of Stakeholders Interested in Joint Capability increases, the Amortized Cost of Joint Capability per Stakeholder decreases, which increases the Stakeholder Confidence in Joint Program Performance, and feeds back to help further increase the Number of Stakeholders Interested in Joint Capability (loop R).

This effect helps the joint program to attract stakeholders. However, while the Number of Stakeholders Interested in Joint Capability grows, the Number of Custom Capabilities Developed also grows, driving up the Joint Capability Complexity and Risk (B2). This depresses Stakeholder Confidence in Joint Program Performance and drives down the Number of Stakeholders Interested in Joint Capability. Similarly, as the Number of Custom Capabilities Developed grows, it also drives up the Schedule and Funding Needed to Deploy Joint Capability, increasing the Gap Between Available and Needed Schedule and Budget (B3). This contributes to Budget and Schedule Overruns, and reduces Stakeholder Confidence in Joint Program Performance. In essence, the loops on the right side of the diagram are driving stakeholders away from the joint program.

However, external to the joint program itself, there is another, larger dynamic playing out in loop B1. As the Defense Budget decreases, there is a corresponding increase in the Consolidation of Multiple Needs into Joint Programs in an effort to reduce costs. This leads to a decrease in the Number of Other Acquisition Programs, and an associated increase in the Competitiveness of the Bidding Process. This in turn increases the use of Underbidding, which reduces the Available Schedule and Budget for Joint Capability when the contract is awarded. Just like the internal pressures of the joint program, this loop also increases the Gap Between Available and Needed Schedule and Budget, again causing Budget and Schedule Overruns, reducing Stakeholder Confidence in Joint Program Performance, and driving down the Number of Stakeholders Interested in Joint Capability. This then increases the Gap Between Needed Stakeholders and Actual Stakeholders (as compared to the defined Number of Stakeholders Needed for Joint Program Success), increasing the Number of Joint Program Failures (since this is a common way for joint programs to collapse—when they no longer have a “critical mass” of participating stakeholder programs), and ultimately undermining the further Consolidation of Multiple Needs into Joint Programs.

“Consolidation into Joint Programs” Scenario Implications for Competition

As the causal loop diagram illustrates, there are two separate dynamics driving the joint program: the exogenous dynamic forming the loop outside of the acquisition program that is driving the competitive pressure that leads to underbidding (B1), and the endogenous dynamic that is an inherent property of the social dilemma2  structure of a joint program that naturally tends to tear it apart (loops R, B2, and B3). In the exogenous dynamic the well-intentioned belief that the consolidation of multiple similar individual needs into fewer joint programs will reduce costs ends up creating a strong incentive for underbidding, creating unrealistic cost and schedule goals that can’t be met, and setting up the joint effort for failure, thus eventually undermining the entire approach of consolidating needs into joint programs. In the endogenous dynamic the problematic reconciliation of disparate joint program stakeholder needs into a single solution, when each stakeholder has an incentive to force the JPO to satisfy their requirements at the expense of the program, has the unintended consequences of increased cost, schedule, complexity, and risk, all contributing again to potential program failure or cancellation. Ultimately, both dynamics work to drive up cost and schedule, thus undermining confidence and eventually driving stakeholders away.

It should also be noted that while it is a point of interest, it happens that the underlying motivation for underbidding is not relevant to the behavior of the exogenous dynamic. Whether underbidding occurs due to honest errors in estimation, or through a deliberate attempt to produce a low bid that will win the contract, the effect is the same. The program will be under cost and schedule pressure from the beginning, which will undermine its performance. However, it should be noted that if low bids were primarily due to honest estimation errors, this wouldn’t explain why underbidding occurs so frequently in acquisition programs—an observation which points to underbidding as an intentional response to the incentives of the acquisition contracting process. A possible explanation for the prevalence of underbidding in acquisition can be found in George Akerlof’s  (Akerlof, 1970) Nobel Prize winning work on “Gresham’s Dynamic,” a corollary to Gresham’s Law.3 Akerlof pointed out that effective but dishonest practices in any business would, over time, become commonplace, and drive out more honest practices. In the same way, once underbidding is employed and found to be successful in winning contracts, accurate bidding will tend to become scarce.

Scenario #2: “Sacrificing Sustainment”

The second scenario presents a dynamic that involves the sustainment planning aspect of software-intensive system development.

“Sacrificing Sustainment” Scenario Narrative

The program was building the next generation of a high-speed communication system to serve a variety of users, and was composed of many different COTS components to leverage commercial technology. As the contractor’s project director described it, “The nature of the design is to be extensible and open, to allow COTS packages to be swapped out as they become obsolete and need to be replaced.” The program was looking into having a government logistics depot sustain the deployed system while the contractor continued to develop the next-generation version of the system.

Hints of Trouble

Planning for sustainment of the system was only one of many responsibilities on the PMO’s plate. People familiar with sustainment knew that “A good sustainment plan will consist of several different plans, including… how to do licensing upgrades, etc.” However, things had not been going smoothly with development, and there was a great deal of pressure on both the PMO and the development team to deliver the new version, pushing longer-term concerns like sustainment to the back. The PMO investigated options to use the development contractor for sustainment, go entirely with a government depot sustainment team, or create a partnership with some combination of government and contractor.

As deployment of the initial system finally began to near, and discussions began on moving its sustainment to a separate government team, the PMO discovered that some key decisions hadn’t even been made internally, much less discussed with the prospective sustainment team. Despite the fact that the use of COTS products was a key part of the development approach, the potential lead for the new government sustainment team at the depot realized that, “We’ve never considered that by centrally buying licenses for COTS products we would be able to get economies of scale, and save government money. From my standpoint, that would just make things harder for us, even though it would save money for the taxpayers.” Additionally, the depot sustainment team pointed out that “We’re still waiting for an updated version of the [PMO sustainment planning] document, which isn’t completed,” and noted that “The [sustainment plan] should have been signed when they went into LRIP4 production two years ago.”

Unintended Consequences

The consequences of a lack of early sustainment planning became more apparent as the program neared completion of the latest release of the production system. Not only were there issues with the sustainment plan in general, but there were fundamental problems on the program side as well. The contractor program manager pointedly observed that the depot sustainment team “…has no capability to sustain [the system], because there is little in the way of a technical data package to hand over” to them, especially in the form of system documentation. The prospective sustainment depot realized that “As far as other things we’d need to have delivered to be able to sustain [the system], we need the source code, the development tools used, and advice from [the contractor] on their experience in building the code.” The contractor was well aware of what was needed to do sustainment properly—but felt no responsibility to supply it as the government had not required it, and had little incentive even  to point this shortcoming out to the government while there was still time to address it.

One of the most serious disconnects occurred when it was realized that the tools the contractor had been using to track installations and versions in the field—not to mention the defect database, the action item database, the engineering drawing database, and 11 other databases—were mostly proprietary to the contractor, even though they would be needed for sustainment. The contractor admitted that “The database is owned by [us], but the information is owned by the government. We hadn’t thought about how to give the government the content in the STR5 database, and there is no government deliverable to turn this over. We could dump it all into hardcopy, but we have no tasking to provide it in a nice, accessible format.” In fact, “If we were asked to pack all of this up, it would take time. It took months to do a conversion via scripts from the old tool to move to the STR tool. Further, even if the government wanted STR, they probably couldn’t buy it” because it’s a proprietary in-house tool that belongs to the contractor. In fact, “There may be 15 databases in total, and you would definitely need all of these, and their files in their office file cabinets, to hand over to a sustainment organization so that they had proper records.”

Even if all of the development tools had been free, or if funding to purchase them were readily available, there was still the issue of the system expertise of the sustainment organization. While very capable in general, most agreed that the depot would need to hire some of the development contractor people with critical knowledge in order to keep them available, and help the depot folks learn how to do maintenance on the system. The contractor sustainment people were “…bright, dedicated… no problems with them. They know all the parts of the system. Anyone else coming in would have to hire those guys because of their knowledge.” However, the contractor was “…already losing people, and they’re behind the power curve in retaining them.” While there was training available for users of the system, “…there’s nothing for maintainers or developers.” The government “…required that a [contractor] retention plan be put in place, to use money they didn’t earn in the award fee to retain employees. But… it’s not enough money to keep 50 people on—it’s only $1,000 per person.”

One of the most important things for a sustainment organization to have when taking on a system is a comprehensive set of internal system documentation—a “technical data package.” As one PMO staff member described it, “For a system to be ready to transition to sustainment I would expect to see well-documented code, all software development folders / notebooks, complete architecture pictures, etc., so you could turn it all over. This is not all available on [this program], because it wasn’t paid for. I do think that [the contractor] has some of these things that were developed on their own nickel, and the government could probably contract to purchase them…” The salient question might be, at what price could it be negotiated?

Contractor Logistics Support

The PMO was disheartened to see the lack of proper preparation that had been made to move the system into sustainment. As one program official wryly phrased the solicitation, “The [PMO] RFI on sustainment said that there are no specifications for the system, but would you perhaps be interested in sustaining it?”

The PMO knew that the development contractor would be very happy to negotiate with the government to offer sustainment services for the system as well—and given their unique qualifications, and the challenges facing any other sustainment organization that might be capable of the work, they were in a very strong (and almost non-competitive) bargaining position.

“Regarding the databases, they had few good answers to the PMO’s question, “Where will the data reside if [the contractor] goes away?”

Locked In

The program office began to realize that the situation they had allowed to happen had neatly painted them into a corner. Regarding the databases, they had few good answers to the PMO’s question, “Where will the data reside if [the contractor] goes away?” The PMO admitted that the contractor folks “…are the only people who can sustain this program, because they’re the only ones who know the software well enough.” While at first it appeared to be a viable solution, it ultimately proved to be infeasible to try to arrange for government depot sustainment when little of the proper planning had been done. It would be too expensive, and too time-consuming to transfer sustainment responsibility to a new organization at this late stage—thus losing the opportunity to compete the maintenance effort.

In the end, there was little surprise to anyone in the program office when they found themselves “…working on the J&A (Justification and Approval6).” When they’d looked at all of the issues with transferring sustainment responsibility, “Because the risk was so high, they decided to go back to [the contractor] as sole source” for sustainment.

 “Sacrificing Sustainment” Scenario Analysis

This scenario is a case of benign neglect on the part of an overworked PMO, and rational self-interest on the part of a profit-seeking contractor. The PMO redirects some of its staff from longer-term activities such as sustainment planning to focus on near-term needs instead. When the program finally nears completion, the government realizes that the lack of sufficient sustainment planning up-front means that they have little ability to competitively and fairly bid a contract for system sustainment. They have no choice but to negotiate additional sustainment services with the development contractor, and so the contractor is now effectively “locked in” for sustainment.

Figure 2 depicts the basic causal interactions and flows of the “Sacrificing Sustainment” dynamic.7

loc-2-figure-2The initial Cost and Schedule Pressure that the program is facing drives a decision: either to choose to invest time and effort to Implement Cost-Effective Sustainment (B2), or to Divert Effort from Sustainment Planning (B1). Either option will, ultimately, help to reduce Cost and Schedule Pressure, but in different timeframes, and with different longer-term consequences. The PMO needs to free up staff to address other near-term priorities, and chooses to Divert Effort from Sustainment Planning (B1). However, after a delay, this means that there have been fewer Actions Completed to Compete Sustainment, which reduces the number of Viable Sustainment Options (R). This in turn reduces the PMO’s Capacity for Competing Sustainment, which increases reliance on Sole Source Sustainment with Development Contractor, drives up Sustainment Cost due to their poor negotiating position, and then lowers the ability to Implement Cost-Effective Sustainment, as the program is left in a sole-source situation with little leverage over the contractor. Without a more cost-effective sustainment alternative, after a delay the Cost and Schedule Pressure increases due to higher-than-expected sustainment costs.

“Sacrificing Sustainment” Scenario Implications for Competition

The outcome for the sustainment approach was the result of a litany of many different issues, with no one problem being overwhelming in and of itself, but collectively producing an inevitable result. One concern is that there is a clear misaligned incentive in encouraging an overworked PMO staff to focus on near-term issues at the expense of longer-term strategic ones. This manifests itself at an individual level when one of the PMO sustainment planners points out that they are making a conscious trade-off between what would be best for them personally (less effort by not centralizing COTS license purchases), vs. what would be best in the longer term for the program and the taxpayers (better negotiating position to reduce COTS license costs). As the causal loop diagram shows, and DoD guidance recommends, making sustainment plans and decisions needs to be done continually from the start of the effort. If such decisions are deferred, they make the sustainment options both fewer and more risky for any choice other than using the original system contractor.

What is clear is that when software development files, test drivers and results, COTS and proprietary software development tools, and other artifacts are not provided to the PMO, it will be costly either to bring on a different contractor for sustainment, or to do it organically. Precisely why this situation originally occurs, however, is less clear. Is it because the PMO doesn’t have the foresight to demand it, or because the PMO staff hasn’t fully thought through sustainment when the contract hasn’t been awarded yet? Is it also partially the result of a “perverse incentive” on the part of the development contractor not to help the PMO make adequate provisions for cost-effective organic sustainment, such as providing full government data rights, and not using proprietary sustainment tools—in order to increase the likelihood of being awarded a contract for sustainment? Regardless of the reason, when the system is fielded, the program office will likely have to continue to work with the development contractor because they are the only ones with the expertise necessary to build and install upgrades to the system. They have inadvertently created a switching barrier—by neglecting to require these things, the government has inadvertently locked the development contractor into doing the sustainment.

To clarify the relationships among all of these ideas, it may be useful to link them together. In the “Sacrificing Sustainment” scenario, there are incentives for PMO management to meet the immediate PMO goals and deliverables with respect to such aspects as finances, contracts, and schedules (i.e., recognition and promotion), as well as disincentives to not meeting them (i.e., damage to reputation). However, the reality is that (as is frequently the case) there is insufficient PMO staff to meet those deadlines. In addition, there are regular 18-24 month rotations of key PMO managers which limit their tenure on the program. Given the conflict between insufficient staffing and looming deadlines, PMO management could choose to acknowledge an impending schedule slip, or perhaps request an increase in the size of the PMO staff. However, given the implicit disincentives to pursuing either of these options (i.e., damage to reputation, perception of incompetence), the actual decision by PMO management is to divert PMO staff from their longer-term sustainment planning to work on the immediate goals and deliverables. This is a classic “social trap” of trading off long-term interests for short-term priorities. The short-term result is that the immediate goals and deliverables are met—but the longer-term results are that sustainment planning is neglected, and as a result the development contractor is locked in for sustainment.

There are various places during the lifecycle of government systems that competition can be used to improve quality and reduce costs.  The use of systems thinking can help to show possible reasons why some opportunities for competition may be lost.  If PMO management can better understand the impact of their early decisions during both contracting and execution, it might be possible to better leverage existing opportunities to introduce competition into sustainment.

Conclusions Regarding Competition in Acquisition

There are some significant conclusions that can be drawn from an examination of misaligned incentives and competition in acquisition.

First, misaligned incentives often occur in the presence of conflicting governance, where governance is the set of rules that control an activity, along with the rewards or punishments for the participants. Unless laws, policies, or other regulations encourage them to do otherwise, people tend to behave in their own self-interest. That said, the existence of an (often unintentional) incentive embedded within the acquisition governance to follow a particular course of action, especially one that is contrary to an individual’s best interests, does not necessarily mean that the incentive will result in that behavior. However, such an incentive does create an ethical dilemma that may pressure people to take an action contrary to their personal interests in order to do what is best for their program, their Service, or their country. Although many people believe they have the integrity to “do the right thing” in such a situation, the point is that acquisition program management and staff should not be put in a position where they must make such a choice, as it ultimately subverts both personal integrity and the organization’s goals. The government acquisition community needs to take steps to adjust governance to better align the personal and program-level goals of acquisition staff with the longer-term objectives of government acquisition. Without this alignment the government acquisition system will continue to rely largely on the integrity of the participants—while at the same time undermining that very integrity.

A second point to be made is that market competition is a long-standing and effective mechanism for achieving efficient outcomes for economic transactions. As a result it is the generally recommended approach for reducing cost and improving quality. However, competition isn’t always as effective as we might like in acquisition contexts. Markets suffer from what economists call “externalities,” where there are side-effects of the transaction that impact others outside the immediate buyer-seller transaction (e.g., air and water pollution). When economists point to markets as being “incentive-compatible mechanisms” (i.e., where participants have incentives to speak and behave honestly), the perceived efficiencies of markets do not account for externalities because those costs are borne by others. Again, the unintended consequences of our decisions and actions often circle back through feedback to affect us in unplanned and unexpected ways—making the study of such “side-effects” vitally important.

Looking at the bigger picture, in scenario #1 on “Consolidation of Joint Programs” the forces of competition contribute to causing a problem by driving contractors toward underbidding with its resultant impacts on program performance. In scenario #2 on “Sacrificing Sustainment” the problem is the lack of competition that could have helped to reduce longer-term sustainment costs. Competition is one of the forces driving Adam Smith’s incentive-compatible “invisible hand” of rational self-interest in markets that moves actors to perform their tasks to our collective benefit, without requiring a sense of altruism on their parts. Competition  is neither inherently good nor bad, but it must be governed and controlled to produce only the effects we desire. This is the beauty of aligned incentives – when they have been properly employed, the behavior we desire will result. The difficulty is in understanding the likely consequences, both intended and unintended, of those incentives when they are part of a highly complex and dynamic system with autonomous actors engaged in multiple levels of feedback from many different sources. The qualitative models presented here are useful and instructional within their limited scope, but they pale in comparison to the complexity present in actual acquisition programs. In fact, the level of complexity and uncertainty in large acquisition programs makes it extremely difficult to identify optimal solutions analytically through such formal approaches as game theory and mechanism design. At the other extreme, it is neither feasible nor affordable to conduct empirical experiments on large-scale acquisition programs in order to fully understand the effects of incentives on behaviors and program outcomes. How, then, are we to make effective progress in dealing with these challenges?

Addressing Misaligned Incentives in Acquisition

In order to make the analysis of organizational problems based on misaligned incentives more tractable, research is being conducted at the SEI to study the recurring behaviors of software-reliant acquisition programs using system dynamics modeling. The following sections describe the direction and status of this work.

System Dynamics

As valuable as the systems thinking approach can be in describing some dynamics, its usefulness declines quickly as the complexity of the dynamic in question grows. Systems  thinking is beneficial in developing a shared understanding of a particular behavior—but this value is lost when those involved are no longer able to fully comprehend the multiple interacting loops of the causal loop diagram. As shown even in the comparatively small scenarios presented here, systems thinking becomes problematic when the complexity of a system makes it impossible to envision how the dynamic will behave. It is at this point that an approach such as system dynamics may need to be employed.

System dynamics is a method for modeling and analyzing the holistic nature of complex problems as they evolve over time (Sterman, 2000). Like systems thinking, system dynamics focuses on capturing the underlying feedback structure of a system’s behavior in an effort to understand which loop’s influence will dominate others over time. System dynamics has been used to gain insight into some of the most challenging strategy questions facing both government and business for several decades now (Sterman, 2000). The System Dynamics Society’s definition of system dynamics states that it is “…a computer-aided approach to policy analysis and design. It applies to dynamic problems arising in complex social, managerial, economic, or ecological systems – literally any dynamic systems characterized by interdependence, mutual interaction, information feedback, and circular causality.”8 As a result, system dynamics is particularly useful for gaining insight into difficult management situations in which well-intentioned efforts to solve a problem can potentially make it worse.

In creating a system dynamics model, all of the elements of the system including such so-called “soft” aspects as policy, procedural, administrative, cultural, and psychological factors along with the more traditional technical factors can be represented and processed. While the results of such modeling produce imprecise predictions of future system behavior, Meadows (Meadows, 1974) points out that “this level of knowledge is less satisfactory than a perfect, precise prediction would be, but it is still a significant advance over the level of understanding permitted by current mental models.” This is precisely the point. Despite the inevitable flaws and incompleteness in any system dynamics model compared to the reality it attempts to represent,  the mental models that are presently the only alternative are substantially worse.

Modeling Acquisition Program Behaviors

The attributes of system dynamics modeling described above make it a natural tool of choice to model the behavior of acquisition programs. We are currently working to model the key dynamic behaviors of joint programs in an effort both to 1) better understand the nature of the dynamics at work that create recurring challenges for such programs, and 2) identify potentially effective mitigation strategies that can be applied to improve program outcomes. In order to create a system dynamics model that is as accurate as possible, groups of experienced acquisition subject matter experts are being used to conduct group modeling sessions to elicit the critical elements of the model, and define the nature of the relationships that exist among the key variables. The model is defined using a commercial system dynamics modeling product that is able to simulate its behavior, allowing it to be validated against both qualitative and quantitative historical data collected from past joint programs.9

The model explicitly addresses both the structural dynamics and the misaligned incentives aspects of the acquisition program being modeled. The activity of the contractor’s software development team represents a set of underlying structural dynamics that both influence, and are influenced by, the incentives that drive the decision-making performed by the key stakeholders of the program. As we have seen in the scenarios presented, key concepts such as fairness, cooperativeness, and self-interest all come into play in these decisions, and can be primary drivers of the outcomes. Once the decision-making models are validated, candidate solution models that apply behaviors to mitigate the problematic dynamics are integrated with the model to evaluate their effectiveness in reducing the behavior. When these trials are complete, if a candidate solution strategy has been validated and shows promise, it may merit a pilot implementation to see if its effectiveness can be reproduced on an actual acquisition program.

It should be remembered that any model of an aspect of the real world will be incorrect because it cannot represent all of the detail involved. This raises legitimate questions about the validity and usefulness of the model. It is certainly essential to formally validate any model that will be used to model behaviors and predict outcomes, but even with these precautions the model will necessarily be incomplete. Although all models are inherently flawed, we still believe they provide important insights, point out discrepancies, and raise valuable questions—and are significantly more useful in understanding complex system behaviors than having no model at all.

Impacts on Acquisition Decision-Making

Once the underlying reasons for counter-productive dynamic behaviors in acquisition programs have been uncovered, acquisition leaders and staff should be educated so that they are capable of recognizing when such situations are present in actual programs, and are aware of the solutions that can be applied to mitigate them. As the Task Force on Defense Acquisition Law and Oversight (“Getting to best:,” 2009) acknowledged, “…the government too often finds itself with minimally experienced and transient individuals leading major acquisition programs…” Because of this it is important to try to significantly improve the decision-making abilities of acquisition program office staff in these areas. In another aspect of the research, this work will use system dynamics models to create interactive classroom games of recurring acquisition behaviors that can provide acquisition staff with an experiential learning environment. This approach will allow them to explore these dynamics and learn about their properties for themselves, which maximizes both learning and retention (Oh Navarro, 2006; Pfahl, Laitenberger, Ruhe, Dorsch & Krivobokova, 2004; Pfahl, Ruhe, Lebansft & Stupperich, 2006), and thus helps to compensate for the present variance in program management office staff experience across programs.

Implications for Acquisition Policy

Using an actual acquisition program (or the entire acquisition system) to test the effects of new policies is not desirable, but has been employed due to the lack of good alternatives. Fortunately, system dynamics modeling provides a feasible, cost-effective way to conduct experiments regarding the efficacy of proposed policies, and can be used to help lawmakers, policy makers, and acquisition professionals come to informed conclusions regarding the relative consequences of different potential decisions. Modeling the acquisition context allows us to create a “testbed” that can be used to help forecast possible specific and broader effects of decisions and actions, and explore ways to improve the overall acquisition process. In addition, it can give inexperienced acquisition leaders and staff the chance to experiment in a safe environment where a) the near and longer-term consequences of actions can be seen both quickly and more clearly, and b) some of the many possible side-effects of their actions can be understood, and thus potentially avoided in actual practice.

Summary

Government acquisition is difficult.  A single large buyer commanding vast sums of money with which to address some of the most complex problems known to us is a unique situation in our society.  Understanding how to improve the process that this single buyer uses to purchase state-of-the-art systems is a daunting challenge. We believe that modeling the incentives and dynamic behaviors of acquisition programs is one of the most powerful and cost-effective ways available to the defense acquisition community of understanding problems, evaluating candidate solutions, and producing better acquisition program outcomes. Even minor efficiencies gained in large acquisition programs from using these methods could produce benefits in functionality, quality, performance, improved time-to-deployment, and direct cost savings that would recoup the investment many times over.  By using systems thinking and system dynamics models, new ways of using competition can be explored in the lab and virtual worlds before trying them out on ourselves.


 

References

Akerlof, G. A. (1970). The market for lemons: Quality uncertainty and the market mechanism. Quarterly Journal of Economics, 84(3), 488-500.

Bennett, J. T. (2009, January 27). Gates outlines DoD acquisition repair plans. Defense News.

Business Executives for National Security, Task Force on Defense Acquisition Law and Oversight. (2009). Getting to best: Reforming the defense acquisition enterprise—a business imperative for change from the task force on defense acquisition law and oversight

Forrester, J. W. (1971). Principles of systems. Pegasus Communications.

Frangos, S. A. (1998). Motivated humans for reliable software products. Microprocessors and Micro-Systems, 21(10), 605-610.

Kadish, R. Assessment Panel of the Defense Acquisition Performance Assessment Project, (2006). Acquisition performance assessment report. Retrieved from website: https://acc.dau.mil/CommunityBrowser.aspx?id=18554

Kim, D. H. (1993). System archetypes: Diagnosing systemic issues and designing high-leverage interventions I. Pegasus Communications.

Madachy, R. J. (2008). Software process dynamics. Wiley-IEEE Press.

McNew, G. J. (2011). An examination of the patterns of failure in defense acquisition programs. Massachusetts Institute of Technology.

Meadows, D. (1974). The limits to growth, second edition revised. Signet.

Novak, W. E., & Levine, L. (2010). Success in acquisition: Using archetypes to beat the odds. Software Engineering Institute, Carnegie Mellon University. Retrieved from http://www.sei.cmu.edu/library/abstracts/reports/10tr016.cfm

Novak, W. E., Moore, A. P., & Alberts, C. (2012). The evolution of a science project: A preliminary system dynamics modeling of a recurring software-reliant acquisition behavior. Software Engineering Institute, Carnegie Mellon University.

Oh Navarro, E. (2006). SimSE: A software engineering simulation environment for software process education. (Doctoral dissertation, University of California).

Pfahl, D., Laitenberger, O., Ruhe, G., Dorsch, J., & Krivobokova, T. (2004). Evaluating the learning effectiveness of using simulations in software project management education: Results from a twice replicated experiment. Information & software technology, 46(2), 127-147.

Pfahl, D., Ruhe, G., Lebansft, K., & Stupperich, M. (2006). Software process simulation with system dynamics—a tool for learning and decision support. New Trends in Software Process Modeling, 18, 57-90.

Sterman, J. D. (2000). Business dynamics: Systems thinking and modeling for a complex world. McGraw-Hill.

United States Government Accountability Office, (2005). DoD acquisition outcomes: A case for change. Retrieved from website: http://www.gao.gov/new.items/d06257t.pd


 

Endnotes

1 Also, Novak, William E. & Williams, Ray C. An Analysis of Recurring Issues Found Across 12 U.S. Air Force Software-Reliant Acquisition Programs, awaiting publication.

2 A social dilemma refers to a class of problems in which the desires of individuals to gain individual benefit or avoid shared costs end up imposing costs on the others around them, making everybody worse off. There is a paraphrased saying attributed to Adam Smith that states this dynamic as, “Individually optimal decisions lead to collectively inferior solutions.”

3 “Gresham’s law” is named for Sir Thomas Gresham, and is a principle of economics that is most commonly stated as, “Bad (overvalued) money drives out good (undervalued) money.” It refers to the notion that when there are two different forms of currency with the same face value, the currency with the higher intrinsic value will drive the currency with the lower intrinsic value out of circulation.

4 Low Rate Initial Production

5 Software Trouble Report—a software defect or discrepancy (bug) report

6  Justification and Approval is used to give the justification for a proposed sole source acquisition award.

7 The “Sacrificing Sustainment” dynamic is an adaptation of the “Shifting the Burden” systems archetype as published in Kim (Kim, 1993).

8 This definition may be found on the website for the System Dynamics Society at http://www.systemdynamics.org/what_is_system_dynamics.htm

9 This work is based on a prior effort using system dynamics modeling to characterize another recurring acquisition dynamic titled “The Evolution of a Science Project,” as described in Novak (Novak, Moore & Alberts, 2012).


Author Biographies

loc-novakWilliam E. Novak is a Senior Member of the Engineering Staff at the Carnegie Mellon University Software Engineering Institute (SEI). Mr. Novak has over 30 years of experience with real-time embedded software product development and business management, and is a researcher, consultant, and instructor in the acquisition and development of software-intensive systems.

loc-levinsonHarry Levinson is a Senior Member of the Engineering Staff at the Carnegie Mellon University Software Engineering Institute (SEI). Mr. Levinson has 25 years of experience delivering high-performance real-time systems in the commercial data communications industry and has full lifecycle experience with systems and software development in application areas including high-speed data communications, multi-processor architectures, and user interface design and implementation.

Comments

comments

Leave a Reply

Your email address will not be published. Required fields are marked *