Project
Measurement Responsibilities
There
are many possible ways of measuring software projects that can
help projects succeed, but there are three overarching
requirements for software projects. They are:
1.
Manage
the project using measures.
Measurement is used to assess whether a software project is
proceeding according to plan, and to inform any necessary
changes if a project deviates from plan. The main data being
tracked are planned and actual effort, schedules, and progress
-- with supporting information about defects, requirements
volatility, and key ongoing activities such as configuration
management.
2.
Provide
data for planning future projects. The GSFC
Measurement Team will use data submitted by multiple projects
team to create estimation and planning models, to analyze how
the Center's productivity and quality of delivered products is
evolving over time, and to assess the effectiveness of
technology development and process improvement efforts. This
data is a small subset of the set used to manage a
project. The GSFC Measurement Program is described here. The Software Process Improvement (SPI) Project is responsible for the operation of the GSFC Measurement Program.
3.
Meet
software measurement requirements as
defined in NPR
7150.2, and requirements of assessment models such as CMMI
or ISO. These requirements have been considered in designing
measurement processes, procedures, templates and this
guidance. Following this guidance and using the processes,
procedures, and templates provided here will help your team
meet these requirements without the agony of reading them in
all their hideous detail. These details can be found here.
These
three measurement responsibilities are defined in the GSFC
Measurement Program Implementation Plan, and are required for Class B projects,
Class C projects, and other Center projects that are estimated at
5 staff years or more effort.
back to
top
Measurement at Project
Startup
The
first five tasks of the Measurement
and Analysis for Software Projects process are performed during
the planning and start up phases of a project:
- Establish
measurement objectives
- Identify the
essential measurement analyses that support these objectives
- Specify the
measures to be collected
- Specify data
collection and storage procedures
- Specify analysis
procedures
The
overall purpose of measuring a project is to be able to
understand how well a project is doing; the guidance below
will help you create a measurement plan that can help you
achieve this understanding. Measurement planning
should be done in conjunction with the rest of your project
planning tasks. In particular, schedule and staffing
estimates need to be documented at the start of a project and
updated as needed to inform the analysis that is done
periodically throughout the project's life.
If
you work in the Flight Software Branch, much of this planning
information has been defined at the Branch level. You merely
need to use it in your Product Plan and use the standard tools
provided by the Branch. For other types of software, the SPI Project has created a number of assets and tools for measurement planning, which are available from the Measurement and Analysis pages of the PAL. In any case, the text below will step
you through what you need to do to create a measurement plan.
-
Locate the correct Software Management Plan
or Product Plan (SMP/PP) template or boilerplate for your
project. The SMP/PP template/boilerplate defines the
elements of a software measurement plan that are
needed. There are several available SMP templates/tools; the
table below specifies which ones you should use:
- In the measurement plan portion of the
SMP/PP template, document the
following:
- The measurement objectives
- The measures that are being used to meet these
objectives
- A summary of how the data is analyzed
Standard measurements can be found in the Measurement
Planning Table, from which a specific set can be
selected for a project. This table includes:
- Measurement area -- These are groups into
which measures are organized to give insight into a
project's health. The specific measurement areas
are defined in NPR
7150.2A.
- Measurement objective -- Pretty much what it
says; what you are trying to accomplish by
monitoring these measures.
- Analysis Summary -- A summary of how measurements in
a particular area are analyzed. For Class B & C
projects, the details need to be documented in an analysis
procedure; see Step #4 below.
- Measures -- A selection of metrics from which
a project can select. Metrics that are required by
the Organizational Measurement Program are marked with an asterisk.
- Collection Frequency -- When and how often the measurement data is collected.
The Flight Software branch uses a tailored version of
these measures; they appear in the Flight
Software Product Plan Template.
The table below highlights supporting documentation
beyond the SMP/PP templates.
-
Set up project measurement collection and
storage procedures. This work entails both
setting up the tools used to collect and store measurement
data and writing the procedures for actually collecting and
storing the data. If the tools already store the data you
need, there is no reason to duplicate your effort. Tools
that will capture information you want to measure
include:
- Discrepancy and change reporting tools
- Point
counting spreadsheets or earned value tools
- Requirements tools
- Project management tools -- Microsoft Project or the
equivalent.
Most of the data entered into measurement tools results
from the Estimation Process and Project
Planning process. Entering this information gives a
Team Lead the ability to later enter actual values for the
same metrics and analyze the project's health by comparing
the actual results to the estimates in the initial
plan. An example of entering planned data would be to
load Microsoft Project with the estimated
schedule.
One change to previous cost estimation approaches is that teams are now required to produce a basis of
estimate (BOE) as part of project planning. In simple
English, this means showing how you arrived at your
estimate. Although the GSFC Measurement Program does
not specify how a team should derive its estimates,
it does require that the team record its method.
Project teams should use this BOE information as a
checklist to make sure that all activities and costs are
accounted for. In addition, the GSFC Measurement Team will
use BOE information to better understand how cost estimates
are produced, to analyze which techniques seem to produce
better predictions, and to feed these techniques back into
standard practices.
The Flight Software Branch has already defined a standard
set of measurement spreadsheets for projects, complete with
instructions on how to load them from other project tools.
Many of these tools, especially Excel, can generate reports
and graphs that can be used in status presentations. Generating periodic status reports will be much easier if you set your tools up well in the beginning .
When writing your collection and storage procedures, keep
them simple. They need to be documented and clear to the
reader, but they don't need to be complex. A good procedure
defines at least:
- Who must provide data?
- Who collects and enters data?
- Which tools are used?
- How frequently is data collected?
For class B projects, CMMI requires highly detailed
instructions for collection and storage. The GSFC Measurement Team has addressed
this developing a Measurement Data Collection and Storage Procedure Template. The template allows you to "fill in the blanks" with details for your particular project, thus greatly simplifying and standardizing the procedure writing.
All procedures should be
documented separately, then referenced in the SMP/PP.
-
Setup measurement analysis and reporting
procedures. This includes setting up analysis
tools as well as documenting the procedures that will be
used to analyze and report measurement data.
For
Class B projects, analysis procedures need to be written in
more detail than is specified in the "Analysis Summary"
column in the Measurement
Planning Table. The elements that are required are:
- Analysis: This element is an objective
evaluation of what the data means.
- Impact: This element describes what impact
(if any) on project cost, schedule or risk is implied by
the data. "No impact" may be a perfectly good answer.
- Corrective action: If the data indicates a
problem, what do you intend to do about this?
For flight software, a standard set of analysis
procedures can be found in the FSB
Measurement, Analysis and Reporting Standard. For other Class B or C projects, a template for the Measurement Analysis and Reporting Procedure is available in the PAL. If you are
running a class D or E project, you may still want to refer
to these procedures where they address the measures you are
collecting.
Setup activities include establishing
tool interfaces. An example of this is defining how data is
transferred from a requirements management tool such as
Requisite Pro into an Excel spreadsheet that is used to
generate graphs for analysis and reporting. In some cases
tools can generate the desired graphics directly; in other
cases transfering data to Excel is recommended.
-
Contact the GSFC Measurement Team to discuss
measurement issues. The GSFC Measurement Team
will provide a point of contact to your project to help with
collecting organizational-level measurements and to provide other
measurement assistance requested by the project and agreed
to by the SPI Project Manager. In the FSB, assistance is
provided via FSB measurement team members.
If you
want planning advice, you can contact the GSFC Measurement
Team before you start your planning work. Otherwise,
contact the GSFC Measurement Team at the end of the planning
process to submit your initial estimates via the Measurement Summary Tool. (See the Organizational Measurement Program) pages for details.
Setting up procedures for
collecting measurements includes:
- Discussing organizational measurement collection,
including a review of the Measurement Summary Tool spreadsheet.
- Defining any needed tailoring of this spreadsheet.
- Providing initial measurement data, including the
basis of estimate for the initial estimate.
The GSFC Measurement Team supports users of all products
accessible from this page, and will also agree to provide
additional consulting if a software team has measurement
objectives beyond the standard set. In the latter case, the Measurement Team
will help a team decide how to define, collect, store,
analyze and report data in support of these
objectives.
back to
top
Measurement during the Project
The last four tasks of the Measurement
and Analysis for Software Projects process cover measurement
activities performed while the project is in progress:
- Collect measurement data.
- Analyze collected data.
- Store collected data and analysis results
- Communicate results to stakeholders
They are carried out as follows:
-
Collect and store measurement
data. This entails
carrying out the collection and storage part of your
measurement plan. In the Flight Software Branch you are
required to use the Branch approach both for planning this
activity and for carrying it out. In other branches, the
responsibility for planning and executing collection and
storage is delegated to the software product
team.
-
Analyze measurement data. The heart of
measurement during a project is the use of the data to
assess a project's state. The analyses listed in the Measurement Planning Table or presented in detail in the FSB
Measurement, Analysis, and Reporting Standard focus on
two main issues.The first issue is whether a project
is progressing as expected with respect to key measures. In
the case of effort, cost, schedule, and progress, this
comparison is against the plan. In the case of quality and
requirements volatility, the comparison is made against organizational models and/or
experience-based expectations.
The second type of analysis is with respect to timeliness
of action. This specifically applies to closing discrepancy
reports and action items. The data will tell a Product
Development Lead whether they are being closed more rapidly
than they are opened and how long they are remaining open.
It is up to the Product Development Lead to interpret
whether or not action needs to be taken to close these items
more rapidly.
-
Re-estimate cost and schedule at milestone reviews or
when re-planning. We strongly
recommend revisiting estimates at major milestones such as
PDRs, CDRs, and test readiness reviews. This is because, at
each subsequent point, the requirements are better
understood and the product has evolved to the point that
more information is available to use in forming an estimate.
As an example, size and effort at CDR can be estimated in
terms of exact counts of new, modified, and reused C++
classes, whereas at proposal time one would be making less
precise comparisons to the features of previous systems. This
estimation table example illustrates the concept of using a different
estimation model at different points in a project; it shows
that estimates start with a high degree of uncertainty but
converge as the actual design evolves. This example is for
FORTRAN ground systems and was published in the early 1990s,
but the concept can be generalized for other domains and
languages.
-
Report on data as part of bimonthly status
reviews. The Branch
Status Review Template defines the places where measures
are to be reported to line management. Typically, metrics
are reported graphically, with a brief (2 sentence at most)
summary of what the data implies to the project. The Flight
Software Branch uses a tailored version of
this template.
The GSFC Measurement Team has a toolset that streamlines the collection, storage, and
reporting of schedule, effort, action items, and other types of data for projects. See the Tools section of the website for a full list of available tools. The analysis of project measures still requires, and
always will require, experienced brainpower.
- Use measurements at milestone
reviews. The Process
Asset Library contains checklists for the contents of
both Project-level and software-level milestone reviews. One
common element is that all these reviews include status
information and estimates of what is needed to complete the
work. The measures collected and analyzed to manage a
software project are precisely the information needed to
present accurate information at these reviews.
In addition to using measurement as a project management
tool, certain measures are required to be provided to the GSFC Measurement Team
using the Measurement Summary Tool , as tailored for the specific project (see
project
startup for details). If the estimates have
significantly changed since the last report, the reason
should be briefly explained on the narrative worksheet. One
should also use the narrative worksheet to enter best
practices that have worked well and should be made known to
future projects, or areas in which the SPI Project should look for
better ways of accomplishing its goals.
back to
top
Measurement at Project Completion
At the end of a software project, the GSFC Measurement Team
will collect a final version of the Measurement Summary Tool spreadsheet. Project completion is defined as
the point at which the final version of the system is
delivered for operations and maintenance. The data collected
includes actual values for the data collected at milestones
(dates, effort, requirements changes, etc.) plus the final
size of the system, however the software team is measuring
it.
The GSFC Measurement Team will use this information to
create planning models that describe the expected behavior of
projects. The narrative information provided by software teams
informs this analysis, and helps set the direction of how GSFC could improve its approach to software development --
both by institutionalizing practices that have worked well and
identifying areas where new approaches are indicated.
back to
top
GSFC & Agency
Measurement Requirements
The material in this section documents the requirements
being addressed by the measurement guidance provided above. If
you follow the Measurement
and Analysis for Software Projects process and all the guidance
provided on this page, you will be meeting the requirements
listed below. The requirements include:
- NPR7150.2A,
which defines measurement requirements
the agency imposes on different classes of software. The
classes of interest to most developers range from A
(human-rated software systems) to E (development support
software). Classes F & G relate
to infrastructure software, and Class H relates to "general
purpose desktop software" such as Microsoft Word.
- CMMI
requirements,
which are levied on class A & B software by NPR 7150.2A.
The NPR requires organizations to reach certain capability
levels in certain process areas; this in turn yields an
additional set of measurement requirements contained within
the CMMI.
- In addition to the above requirements, the GSFC
Measurement Program Implementation Plan defines data collection
requirements. For Class B and C projects, these measures are
a subset of what is required by the NPR and CMMI. The NPR
does not define any measurement requirements for Class D and
E projects, however if they are large enough (estimated
effort of 5 or more staff years) a small set of data is
required at major milestones, and this guidance recommends
more frequent collection of effort, progress and defect
data.
At this writing, the GSFC Measurement Program Implementation Plan does not
address Class F, G, or H software.
back
to top
|
