Reinventing IT Test and Evaluation
Written by Dr. Steven J. Hutchison
DISA has undertaken several major projects to improve
how DoD acquires and tests information technologies.
We are organized much the same way as service test organizations, with a headquarters element—the T&E Management Center (TEMC)—responsible for strategic T&E resource management, and an operational test agent—the Joint Interoperability Test Command (JITC)—responsible for test execution and interoperability certification.
I have previously written about the changes to DISA’s T&E structure, but it’s not about the organization—it’s where we’re going that is making a difference. (See MIT 11.8, September 2007.)
The DISA TEMC and JITC are taking on new challenges and providing new leadership in helping DoD achieve our vision of agile, responsive and warfighter-relevant T&E that is an enabler of improved acquisitions.
I would like to see the Department of Defense adopt a new approach for acquisition and T&E of information technologies. Our IT programs struggle to be agile under the current acquisition processes. There is much we can learn from industry in terms of delivering IT capability. For the commercial sector, time to market is critical to the bottom line; for DoD, time to field is critical to war fighting.
We have undertaken several major projects to improve how we acquire and test information technologies in the department.Mp> We have engaged the National Academies of Sciences to conduct a study on improving policies and processes for acquisition and testing of information technologies in DoD. The study is a comprehensive assessment of legislative and regulatory requirements, processes and capabilities in the commercial IT sector, various concepts for systems engineering and testing in virtual environments, as well as challenges of the acquisition culture in DoD. The study committee will conclude and publish its findings later this year.
We’re not just studying the problem; we’re implementing change. Last year, a combined industry/government council on OT&E recommended sweeping changes to the format and content of the T&E Master Plan (TEMP). During the committee discussions on the proposed TEMP, it was clear that everyone thought the ideas were good and should be pursued. Unfortunately, no decisions were made to experiment with the format, and it looked as if the good work of the group was going to fall by the wayside.
DISA volunteered to use the new TEMP for the Net-Centric Enterprise Services program, since a new TEMP was required for the pending Milestone C. For comparison purposes, the Milestone B TEMP exceeded 360 pages, required 18 signatures and took over six months to obtain approval once it began the staffing process. The new document prepared for Milestone C, which is generally a far more detailed plan, took only 70 pages, 14 signatures and 90 days from the beginning of document preparation to final approval. That’s a successful pilot by any measure.
DISA’s leadership has been essential in another critical area—testing in the joint mission environment.
CHALLENGES AND OPPORTUNITIES
Change doesn’t come easy or quickly in DoD, but there are major shifts occurring in the department today that have significant impact on T&E. They include shifts from requirements to capabilities, from service-centric to “born joint,” and from traditional to rapid acquisition.
For the T&E community, these changes bring new challenges; for DISA, they present new opportunities for engagement and leadership.
Within the T&E community, new energy is being applied to the Testing in a Joint Environment Road Map. The road map describes actions needed to achieve objectives described in the Strategic Planning Guidance for FY06–11 to “ensure that test and evaluation is conducted in a joint environment and facilitates the fielding of joint capabilities.”
DISA is a staunch supporter of the road map and is a key player in its implementation. One of the key initiatives of the road map is the Joint Mission Environment Test Capability (JMETC). For many years, DISA managed a program known as the Joint Distributed Engineering Plant (JDEP), which had most of the essential ingredients of what was envisioned for JMETC. When DoD looked to us to help advance the road map and the JMETC, we agreed to transition JDEP to JMETC. This involved the transfer of program management and, not insignificantly, program funds as well. With our support, the JMETC is moving forward as a critical infrastructure component envisioned in the road map.
Another major road map initiative is the development of methods and processes to test and evaluate systems in a joint mission environment. To address this, the director, operational test and evaluation chartered a three-year project known as the Joint Test and Evaluation Methodology (JTEM). The purpose of JTEM was to develop methods and processes for creating realistic, distributed, live, virtual and constructive joint test environments to evaluate system performance and joint mission effectiveness.
DISA has been contributing to JTEM since its inception and currently provides the test director. JTEM focused on how to design and execute tests of a complex system of systems in the joint mission environment and published guidelines in a document titled, “Capability Test Methodology” (CTM). Since all of our T&E activities in DISA are joint, we have made it an objective to leverage the CTM (and the JMETC when possible) in all future testing, not just to improve our own processes, but also to provide feedback to the department for future refinement of these guidelines. In fact, DISA will conduct the first formal CTM-based OT&E this fall.
SOFTWARE DEVELOPMENT
Once upon a time, DoD acquired IT following a different process from weapons systems. In 1996, the department merged the acquisition of automated information systems into the DoD5000 series. When you think about that, the IT revolution was in full swing in 1996, with major online businesses, such as eBay, Amazon and Google, emerging as IT leaders. Given what emerged in the commercial sector in the decade-plus since, hindsight might suggest that DoD’s “one size fits all” model has just about eliminated innovation in approaches to acquisition and results in much of the department’s cost overruns, poor performance and lengthy schedule delays. In fact, a recent analysis of major automated information systems shows that the average time from program start to initial operational capability, following the DoD5000 model, is 91 months.
In the commercial IT sector, innovation has produced numerous approaches to software development. A quick search of “software development process” at the online Wikipedia site highlights several approaches, including “waterfall,” “extreme programming,” “Agile,” and others. Agile is one of the fastest growing and most productive approaches to software development.
I will not go into detail here, as there is a wealth of information about Agile on the Web. Fundamentally, though, Agile is about teamwork and short, iterative development cycles, sometimes called “sprints.” In an Agile development environment, testers and users are part of the team formed to deliver working capability in short iterations, which are typically 90 days or less.
Testing is another part of the acquisition life cycle impacted by overly prescriptive process. In the DoD IT T&E community, we have four different test activities that occur in each development cycle, with different test organizations each vying for time to conduct their events, under different test conditions, for different decision-makers.
The four T&E activities are developmental testing, operational testing, joint interoperability test and certification, and information assurance certification and accreditation. Unfortunately, we don’t always conduct these test activities in an integrated manner; nor do we combine results in evaluation reports, and this can mean that different information is being presented to the decision-makers—the milestone decision authority, interoperability certifier and the IA certifier, or “designated approving authority.” Acquisition decision-making would be greatly improved if the various T&E activities were synchronized to produce a single evaluation that satisfies the needs of all three decision-makers.
Actual test execution can be a time-consuming process. The scope of most test events are considerable and reflect the magnitude of development effort required in the capabilities development document (CDD). Activities that occur in the test execution phase include test plan approval, training, the four test activities and reporting. This test execution phase typically exceeds six months. This approach to T&E clearly would not support the Agile development process, nor can I imagine that any commercial entity would pay a test agent to do T&E the way we do it. But I could be mistaken.
Agility is of course a major goal in IT acquisition. Agility must be part of the requirements process, development, testing and oversight. Agility in the requirements process means down-selecting from the work stack just those items of immediate importance. For us, we should never expect a program to deliver everything in a CDD in the first increment. Instead, we must prioritize the requirements within the CDD, and focus the iterations on the immediate needs of the warfighter—the one or two capabilities that will have immediate beneficial impact if fielded.
The agile PM has to transition from the waterfall approach to a highly collaborative environment where all stakeholders work together to develop, test, fix and verify solutions. In DoD’s evolutionary acquisition waterfall process, each increment is guided by the JROC, and managed to the same set of milestones. This is not an agile process.
For IT capabilities, DoD could modify its acquisition process to reflect greater agility as seen in the commercial sector. In this model, development occurs in short duration iterations, focused on high priority warfighter needs. The process begins with a tradeoff analysis to determine the best approach to satisfy the user need. In this phase, the capability portfolio manager determines if a capability already exists to adopt for the enterprise, a commercial solution exists to buy, or no capability exists and so a new development effort must begin.
Once the acquisition approach has been determined, the effort shifts to integrating the new development effort into the capability architecture. The development process focuses on high-priority goals for each iteration. At the end of each iteration, working capability is fielded, while non-working software is returned to the development effort for re-work.
For T&E to be an enabler in such an environment, testers must be equally agile. Long lead time test plan approvals and lengthy staffing of test reports cannot be the standard, or T&E risks becoming an obstacle rather than an enabler. The approach I recommend is designed to achieve this objective. It is an integrating model where activity by one is accepted by all.
I refer to the model as capability test and evaluation (CT&E), and I describe it this way: “one team, one time, one set of conditions.” The intent of CT&E is to focus the four T&E activities on the goals of that iteration, and merge them into one short duration test period.
CAPABILITY TEST AND EVALUATION
First and foremost, CT&E is about teamwork. In this age of resource constraints, we should view all test events as a shared resource. Every event should be an opportunity for the test community to collect data to evaluate the capability. Secondly, CT&E seeks agility, and finally, CT&E is also about sharing information.
If we treat each event as a shared resource, we will obtain three immediate benefits:
- Focus on the critical needs of the warfighter
- Elimination of unnecessary duplication in test activities, potentially lowering cost and shortening schedule
- Improvement in the evaluation of capabilities and limitations, thereby improving risk management and decision-making.
In other words, the objective of CT&E is agile acquisition of enhanced capabilities for the warfighter.
So how do we do this? To bring all these test organizations together, we need a responsible test organization (RTO) to be in charge. The milestone decision authority should designate the RTO. Once designated, the RTO assembles a capability test team (CTT) consisting of empowered representatives of each test organization. The CTT plans and conducts CT&E events.
The CTT members coordinate the details of test execution so that each test organization gets what they need from the event. Developmental testers obtain information about critical technical parameters, operational testers can address key performance parameters and critical operational issues, and the interoperability and information assurance testers obtain the data they need to complete their certification responsibilities.
The CTT must have access to the right tools and instrumentation, as well as access to users, readily available test scripts, and the ability to quickly analyze test data. It is important that the conditions of test be as operationally realistic as possible or the event may not satisfy OT&E considerations, and the idea is not to have to do tests more than once. Typical users have to be involved to exercise the capability under test; this is not unlike beta testing in the commercial sector.
Just as having to test more than once is not desirable, writing more than one evaluation report—meaning one report for interoperability, another for information assurance and yet another for OT&E—is equally not desirable, and can result in good information and analysis being provided to one decision-maker but not to another. In the CT&E model, the CTT writes one report for use by all decision-makers.
In DISA, we are moving to improve how we test joint in an agile, responsive manner. The DoD IT acquisition community needs an Agile 5000, but we do not have to wait for it. As our acquisition programs migrate toward agile development and test, the acquisition and T&E community has to be ready.
Likewise, we need agile oversight. We can improve the way we acquire and test information technologies by focusing on the immediate needs of the warfighter and development in short duration sprints. In T&E, a one-team, one-time, one-set-of-conditions approach is essential to delivering improved capabilities in significantly less time. ♦
_________________________________
Dr. Steven J. Hutchison is DISA’s test and evaluation executive.
