What is the currently known effectiveness of the defect detection process before release?
Total defect containment effectiveness = prerelease defects / prerelease defects +
postrelease defect
What is the currently known containment effectiveness of faults introduced during each constructive phase of software development for a particular software product?
phase I errors
Phase containment effectiveness for phase I = phase I errors / phase I errors +
phase I defects
Using the G/Q/M approach gave Motorola the opportunity to clarify the semantic use of specific terms such as errors versus defects and the boundaries of given development phases while formulating questions and metrics. (For purposes of clarification, an error is a problem found during testing of the phase in which it was introduced, and a defect is a problem found later than the phase in which it was introduced.) Managers were able to use the information generated from the measurement program to pinpoint errors and defects by software development phase. This information helped them identify areas in the software development process that required changes and to make the needed changes based on information instead of intuition.
[4 ]M.K. Daskalantonakis, “A Practical View of Software Measurement and Implementation Experiences within Motorola,” IEEE Transactions on Software Engineering, 18, no. 11, 998–1010, 1992.
WHAT CAN A MEASUREMENT PROGRAM MEASURE?
The complexity of developing and maintaining business systems makes it difficult to isolate the activities and areas for which goals, questions, and metrics should be formed. The balanced scorecard approach to general business measures encourages managers to expand beyond traditional financial metrics and more thoroughly integrate organizational strategy with resulting performance.[5]
The balanced scorecard defines four different perspectives for performance measurement:
§ Financial — How do we look to shareholders?
§ Internal business — What must we excel at?
§ Innovation and learning — Can we continue to improve and create value?
§ Customer — How do our customers see us?
Although these perspectives were originally defined for measuring an entire organization, they can be translated to the IS world. There they, respectively, are the project, product, process, and performance perspectives.
[5 ]R.S. Kaplan and D.P. Norton, “The Balanced Scorecard: Measures That Drive Performance,” Harvard Business Review, January-February, 71–79, 1992.
PERSPECTIVES OF PERFORMANCE MEASUREMENT
Exhibit 1 depicts the importance of the four perspectives to IS measurement. They are discussed in-depth in the following sections.
Exhibit 1. Balanced Scorecard Framework for IS Measurement
Project
The objective of the project perspective is to understand and measure the characteristics of a specific development or maintenance project by focusing on the attributes that make each project unique. Many organizations use a development or maintenance project as the vehicle to gather data concerning personnel effort and cost s. Within the project perspective, an IS organization has the opportunity to use this data to create metrics that provide insight into how funds for software development and maintenance are used. What is measured depends on the scope of the project. The at tributes of a given project provide information about such factors as personnel utilization, project estimation, and the nontechnical activities associated with a project.
Product
This perspective emphasizes the attributes that differentiate a product from others.
The metrics applicable to this perspective are used to understand the growth and progression of a development and maintenance product. It is important to understand the internal scope and attributes of each specific product that composes a system. Managers can use this information to devise new testing procedures, determine individual product defects, or improve the accuracy of product estimation.
Because a product frequently lives longer than the project used to create it, product data is gathered over the life of a product. Information generated from the data helps a manager better determine the life expectancy of a given product.
Process
The Software Engineering Institute (SEI) Capability Maturity Model has brought renewed focus to the software development and maintenance process. The process perspective highlights the desire to modify the process used to develop and maintain information systems so that procedures reflect the best practices discovered in the industry and within a given organization. The measures of this perspective consider organizational and human social interactions, as well as the methodological and technical implications of the development/maintenance process.
Performance
The performance perspective measures the outputs of an information system. It encompasses measurements that track both the traditional technical measures of performance as well as metrics that indicate the success of the system as defined by the strategies and policies of an organization. Defining this perspective requires defining system success.
Importance of Balancing the Scorecard
Using the balanced scorecard approach helps IS managers ensure that all aspects of the IS function are appropriately represented in the measurement program. Instead of focusing on a single area, such as personnel productivity for a specific project, the balanced scorecard provides structure from which to view each perspective of IS
operations.
Using the scorecard thus helps IS managers delineate critical areas and define appropriate goals, questions, and metrics for each of them. The importance of the perspectives varies across organizations. For example, one company may be more interested in the performance of IS as a whole, whereas another might be more concerned about the productivity of project development activities. The balanced scorecard framework does not attempt to dictate the relative emphasis on each area but instead serves as a guideline for managers during development of specific measurement goals.
Sample Data for IS Metrics
Once an organization decides to implement a measurement program, the problem is usually deciding what not to measure, rather than deciding what to measure. Many organizations collect data for so many different metrics that participants in the measure ment program become cynical and the effectiveness of the program is greatly reduced. In other organizations, the problem is to determine what kinds of data are available to be collected. Exhibit 2 lists categories of metrics for each perspective of the balanced scorecard framework, data for potential metrics in each category, and sample metrics. The exhibit is not comprehensive but rather an abbreviated list of possibilities for each perspective of the framework.
Exhibit 2. Metric Categories, Data, and Samples for the Four Perspectives of IS Performance Measurement
Category
Sample Data
Sample Metrics
Project Perspective
Financial, type and
Total estimated and actual time Cost per function
scope
and estimated and actual cost
point (FP)
per predefined project activity,
type (e.g., development,
maintenance), estimated and
actual project function points
Personnel
Experience level, experience
Productivity
type and education of personnel, ratings: time/FP
years using a specific
for different levels
development environment,
of experience and
number of contractor personnel, education
number of employees
Methodology
Type(s) used, level of
The metrics for
automation, testing techniques, methodology are
number of models
summarized for
the entire software
process rather
than for a
particular project
Interface
Number of meetings, meeting
Percent of time in
type and length, number of
meetings and by
requirements and design
meeting type
Category
Sample Data
Sample Metrics
changes, pages of
documentation, hours of
customer training
Product Perspective
Financial, type and
The same data and metrics used
scope
for a project are also applied to
a single product; one project
could result in many products, or
it might take many projects to
produce a single product;
product measurements exist
over the life of the product,
whereas project metrics are
closed out when the project is
completed
Quality
Number of defects and errors,
Number of
number of test cases, number of defects/FP
change requests, number of
changes, amount of usage,
complexity rating, number of
reused modules, number of
support calls
Results
Business objectives translated
Inventory percent
into quantitative goals
level
Efficiency
Amount of memory, disk,
Average/peak
processor cycles, response time, response time
operator time
Process Perspective
Organization
Number of general meetings,
Maturity level
type of meetings,
assessment
communication methods, hours
by activity; amount of office and
desk space
Personnel
Data in addition to that gathered Metrics in addition
for a given project includes
to those listed for
vacation days taken, vacation
a given project
days worked, number of working
days, number of employees,
number of contractors, number
of training days
Methodology
Gathered by project
Overall
productivity by
methodology
Performance Perspective
Satisfaction
User satisfaction survey, number Average time to
Category
Sample Data
Sample Metrics
of system users, number of
respond to help
workstations, number of reports requests
generated, number of reports
used, number of screens,
number of screens viewed,
number of help requests
Integrity
Number of errors discovered
Average number
after delivery, number of errors of errors
discovered within a selected
(classified by
period of time, error severity
severity)
discovered after
delivery
As the exhibit indicates, there are more types of data than there is time available to collect them. A measurement program would not be cost-effective if the data necessary to produce all interesting metrics was collected. The best method is to focus on basic metrics such as size, defects, effort, and user satisfaction before moving on to other metrics.
Under the G/Q/M approach, the choice of metrics follows the definition of goals and formulation of specific, answerable questions. To achieve a balanced scorecard of measurement, IS managers must ensure that the metrics selected span each of the four perspectives. If one perspective is not measured, a distorted picture of IS may be the result of the measurement program. Because the balanced scorecard framework is used as a guideline to pinpoint areas for control and subsequent improvement, the implications of such a distortion are enormous.