The project was managed essentially on a contract basis — there was little description of specific requirements for product releases, so programmers worked as time and materials were available and the project continued as long as management was satisfied with the progress. Releases occurred often and were not well controlled; access to programs was controlled even less.
In early 1990, Motorola began asking local software contractors to conform to sector and corporate requirements for higher-quality software development. Because PIMS
was of a magnitude and importance that it could have significant impact on Motorola manufacturing, requirements were specifically placed on that project. Unfortunately, few experts from IS were available to help guide process improvements. An initial process plan was developed by Sterling and the FMG, which helped to lead the evolution from an informal, ad hoc environment to a more formal process.
About the same time, an initial, informal Software Engineering Institute (SEI) assessment had been made of the FMG CIM Development Group that was overseeing the PIMS project. This assessment indicated many deficiencies in the development processes used both internally and in contracted work. The internal recognition of the need for significant software process improvement was strong, especially for large projects such as PIMS. By then, the factory was depending on the complex PIMS system, which was more than 100,000 lines of code in Tandem COBOL and SQL. (It is now about 250,000 lines excluding libraries.) Quality was critical.
EVOLUTION OF THE PROCESS
Once the need for higher software development quality was ackowledged internally, improvements to the process were made in several areas. Following is a brief overview of the status of many of the improvements that have been made.
Project Management
Project Planning. Besides the strategic (one to five year) planning that takes place, specific release planning is continuously performed for PIMS according to business and manufacturing needs. A balance is maintained between enhancements that help current and future users and features affecting ongoing product support, such as data archiving. All these needs are weighted against budget and time constraints.
Preliminary estimates of budget and schedule for a release are made, then updated as functional analyses are performed. Bull’
s-eye charts showing schedule and budget
variation are used to track progress to the release estimates (Exhibit 1) throughout the course of a version.
Exhibit 1. Bull’
s-Eye Schedule Chart
Managing the Life Cycle As the process grew more formal, it became apparent that a new life cycle model was appropriate. Rather than a spiral model, the more formal modified waterfall life cycle for each release was adopted (see Exhibit 2). In the PIMS modified waterfall life cycle, a number of enhancements are targeted for each version release. Each enhancement undergoes a detailed functional analysis that includes a current system analysis, functional requirements, design options, recommendations, and a work plan. Only after this analysis has been completed is an estimate of implementation work given for that enhancement or group of enhancements; a well-documented design is then performed. Ongoing meetings of the design team, as well as design review with users, support staff, and other developers, ensure that the final sign-offs are usually a formality. After coding, the software undergoes three tiers of testing to ensure system reliability, functional correctness, and performance.
Exhibit 2. PIMS Modified Waterfall Life Cycle Model
Because of the increased complexity and size of the PIMS project and the larger project team involved, a modified organizational structure was necessary to effectively handle all the project responsibilities and variables. A more effective, cross- functional development team structure was put into place. Detailed resource tracking charts were used to help allocate programmer/analyst resources, as well as overall resource balancing and budgeting. The Motorola project manager remains responsible for long- term strategy, overall budget and planning, business and project coordination issues, leading improvements in the development process, and coordinating efforts with the Motorola implementation team.
Configuration Management
The complexity of the project and the product also demanded much more formal program configuration management than was necessary when only a few people were working on a few thousand lines of code. However, there was no configuration management system or tool available on the Motorola Tandem system. Therefore, the PIMS team created both a process and an integrated tool to support configuration management and change tracking.
The PIMS program management system provides for work request tracking, security, and change management. All work requests (e.g., proposed enhancements and defect reports) are tracked through an automated system. All program files are secured to prevent unauthorized modifications. When work is to be performed, a program is signed out using a work request reference number, and only the person who signed out the program is allowed to work on it. After modifications are made and the program passes unit quality assurance (QA), it is signed back with the previous version saved. Sign-out and sign-back procedures update work request status, and every PIMS program maintains a complete change history.
Structured Requirements Analysis and Design Much more discipline has been adopted in the analysis and design of enhancements since the early days of the project. Design options are iterated and reviewed in early project phases, and complete documentation is maintained. Design — and design reviews — are viewed as a process rather than an event. Because reviewers attend design meetings throughout the creation of a design, they can understand subtleties often impossible to catch in one or two review sessions. Data flow diagrams, data structure diagrams, and program structure charts are used to capture and communicate analysis and design ideas. Entity–relationship diagrams of the overall relational table structure are maintained. All these are used for formal design reviews with support staff and key users before sign-offs are obtained.
Development Tools
The Tandem development environment is an extremely difficult one in which to create a large, complex system because there are no integrated computer-aided software engineering tools and few development support tools available. Most of the standard development support tools available in other environment either do not exist or are not available in the Motorola Tandem environment. To increase the quality and productivity of the process, unique and custom approaches have had to be developed. For example, the PIMS team developed the Tandem online tracking database in SQL to track bug reports and enhancement requests that drive both project metrics and future development. In addition, the automated configuration management tools described earlier were developed internally.
Testing and Acceptance Criteria
All programming work goes through a three-tier testing process. First, a software QA test of each new or modified program unit is performed by a PIMS developer other than the author. These tests are based on unit test plans that are documented for use on future releases. When all enhancements for a given release are completed, system integration testing is performed to exercise interfaces between all programs and to verify the integrity of overall database designs. Finally, user testing based on predefined functional test plans is completed to ensure functional and regressional integrity. Before any version is released to production, it is installed on a separate Tandem test pathway (i.e., a nonproduction mirrored system partition) for verification in a realistic environment. Metrics on testing are collected throughout the system and user testing tasks. Bug-detection rates help determine testing progression and provide insight into how clean the releases are (Exhibit 3).
Exhibit 3. System Test Metrics (Cumulative Test Hours versus Bugs Reported and Fixed)
Metrics
After the process had evolved to a fairly stable level, metrics were needed for better understanding of the process itself and the improvements being attempted. An automated database for metrics data collection has been developed. It includes problem report entry screens for key problem variables, as well as status tracking of open problems. The severity of each bug is ranked, and separate trends are maintained for what are ranked the major and medium problems versus the minor ones. The product quality metrics currently tracked (Exhibits 4 through 8) are as follows:
Exhibit 4. Released Software Quality Metric
Exhibit 5. Customer-Found Defects Metric
Exhibit 6. Post- Release Problem Report Activity
Exhibit 7. Post- Release Problem Report Aging
Exhibit 8. Cost to Fix Post- Release Problems
• Released software quality: The total released number of defects per 1,000
assembly-equivalent lines of code (by release version).
§ Customer-found defects: The total customer-found number of defects per 1,000 assembly-equivalent lines of code (by release). This and the released software quality metric target the Motorola corporate goal of a Six Sigma quality level.
§ Post-release problem report activity: The number of newly opened and total open problems by month.
§ Post-release problem report aging: The mean age of open pro blems and the mean age of closed problems per month.
§ Cost-to-fix post-release problems: The total billed cost spent fixing previously released problems each month.
These metrics give Motorola the best view into items critical to both end users and project managers. The project team plans to expand metrics collection and tracking as it determines other measures that would indicate software quality and project success.