WHAT’S YOUR TESTING ROI?
Have you ever thought about how expensive software testing is? Even if
your testing is outsourced, testing is an expensive, time-consuming
activity. Like coding, testing is labor-intensive. Tests have to be
planned, developed, debugged, maintained and executed. Defects, once
found, need to be tracked and triaged. Decisions need to be made about
which defects should be fixed. When fixed, defects need to be verified and
regression testing needs to be performed to determine if fixes have
introduced new defects.
Like most other expensive, time-consuming business processes, it’s
important to know what the Testing Return on Investment (ROI) is and how
it could be improved. Many software companies have no idea how much
testing costs, much less their Testing ROI. You’d think that an activity
so critical to the business and so expensive would be constantly measured,
monitored, and managed...
WHAT IS TESTING ROI?
The most basic definition of ROI is:
ROI = [(Payback - Investment)/Investment)]*100
where: Payback is the amount earned from your investment
Investment is the cost of resources required to generate the payback
To calculate Testing ROI, we need to understand both the investment and
the payback.
THE INVESTMENT
Testing is performed by developers (unit testing) and QA (functional,
system, validation testing are several terms used for testing performed by
QA). For this discussion, we’re focusing only on testing performed by QA.
The testing investment can be determined by looking at testing costs, such
as:
- Cost of resources (labor) spent on all testing and testing-related
activities
- Cost of resources (equipment and infrastructure) to support testing
activities
- Costs of resources (labor) required to deal with defects found from
testing
Companies may have additional costs that may be specific to their
business. Quantifying these costs is certainly feasible.
THE PAYBACK
The payback from testing is determined by looking at:
- Customer satisfaction is higher as a result of encountering far fewer
defects
- Employee satisfaction is higher as a result of greater pride in
producing higher quality products
- Sales and revenue are higher because higher quality products compete
more successfully against products perceived to be of lesser quality
- Development and support costs are lower because higher quality products
are less expensive to develop and support
- Maintenance costs are lower because higher quality products are easier
to maintain and enhance
- etc...
Capers Jones [1] surveyed hundreds of software development companies. Of
those that produce higher quality software, he found that they have:
- shorter development schedules
- lower development and maintenance costs
- better reliability and longer mean time to failure
- larger market share
- better user satisfaction
- better employee morale
- less chance of ending up in court
Quantifying testing payback is difficult, not because we can’t identify
what it is, but because it is difficult to measure. We all know that
testing identifies defects and that there’s a strong correlation between
defects and quality. When defects are fixed properly, quality improves.
As a result, it is difficult to measure the Testing ROI because many of
the Paybacks are not conducive to measurement or represent intangibles.
For example, if you are among the few software companies who measure
customer satisfaction, can you determine what portion of customer
satisfaction is attributable to product quality? Can you determine what
portion of product quality is attributable to testing (as opposed to using
good engineering practices)? This is hard to do...
So, while we all agree that testing is a good thing to do, we can’t
realistically measure the Testing ROI. But, since we all agree testing is
a good thing to do, we can identify ways to make testing more effective.
Taking steps to make testing more effective will clearly improve the
Testing ROI, even though we may not be able to directly measure it.
Here are some examples of what I mean when I say we can improve testing
effectiveness:
- Are you testing the most important features? How do you know?
- Are there features you’re not testing? I bet you are...
- Are you testing the same features many times? I bet you are...
- Are you testing against vague and ambiguous requirements? I bet you are...
- Does your regression suite have tests that are not relevant? I bet it
does...
THE PROBLEM
Testing requires a significant investment in time and effort. Each year,
companies spend hundreds of thousands of hours testing software. Typical
test suites often number into the thousands of tests, many of which
require hundreds of hours to develop, maintain, and execute. Often, tests
are written against requirements that are vague and ambiguous. Regression
test suites evolve over time and often include large numbers of what I
call “non-productive” tests. These tests often are looking for problems in
areas where there aren’t problems. Couple this with the fact that testers
are inclined to develop more tests in the areas of the application they
are most familiar with, leaving other areas under-tested or not tested at
all.
We need ways to assess testing effectiveness. And, we need to assess
effectiveness BEFORE we start testing so that we increase the Testing ROI.
MEASURING TESTING EFFECTIVENESS
Very few techniques have been developed to measure testing effectiveness.
Here are some examples of what has been done:
Chernak [2] defined a measure called test case effectiveness (TCE) as:
TCE = Ntc / Ntot * 100%
where:
Ntc = defects found by test cases
Ntot = sum of defects found by test cases and found by other means
Huber [3] at Hewlett-Packard developed a set of metrics
(http://www.swqual.com/articles/HPMetricDescription.pdf) to measure test
effectiveness. The H-P metrics illustrate the importance of measuring the
effectiveness of the testing activity.
FACTORS THAT IMPACT TESTING EFFECTIVENESS
In looking at the testing effectiveness measures from Huber and Chernak, I
think we need to focus on productive vs. non-productive testing as one way
to improve testing effectiveness.
Productive tests are tests that have a much higher probability of
finding defects. Since the whole purpose of testing is to find defects,
we would obviously want to develop as many productive tests as possible.
Non-productive tests are tests that will likely never find a defect, no
matter how many times the tests are executed.
If you accept the premise that we need to focus on developing as many
productive tests as possible and that having more productive tests will
surely improve testing effectiveness, we can then focus on identifying
factors that affect our ability to write productive tests. My list
includes:
- Testing against Poorly written requirements
- Unrealistic development and testing schedules
- Lack of measurement capability
- Lack of testing infrastructure (staff, hardware, systems, automation
tools)
- Lack of domain knowledge
- Lack of training in basic testing techniques
Given these factors, we can now look at ways to assess Testing
Effectiveness.
TESTING EFFECTIVENESS ASSESSMENT
A Testing Effectiveness Assessment is a process that can improve the
Testing ROI and includes the following five steps:
1 Scrub the Requirements
Tests can only be as good as the requirements they are intended to test.
If requirements are vague and ambiguous, the tests will not be very
effective. Therefore, the first step in the assessment process is a
thorough review of the requirements.
Based on this review, a Requirements Scrub may be necessary. For more
information on writing requirements that are unambiguous, check out the
Sept 2004 Food for Thought
(http://www.swqual.com/newsletter/vol1/no1/vol1no1.html).
2 Update the Requirements Trace Matrix (RTM)
Once the requirements have been scrubbed, the next step is to review and
update the RTM. The RTM is an extremely important document. In its
simplest form, it provides a way to determine if all requirements are
tested. However, the RTM can do much more.
In this example, the RTM is used as a test planning tool to help
determine how many tests are required, what types of tests are required,
whether tests can be automated or manual, and if any existing tests can
be re-used. Using the RTM in this way helps ensure that the resulting
tests are most effective.
The RTM can serve many purposes over the course of a development
project. Initially, it can be used as a planning tool (as illustrated
above). Once the tests are developed and Validation Testing has begun,
the RTM can be used to help determine the extent of regression test
required based on the relationship be requirements, design, code, and
tests as illustrated below.
When it becomes necessary to perform regression testing, the accurate
information included in the RTM will be invaluable in helping to select
a reasonable set of tests to run.
3 Analyze the Existing Test Suite
The RTM can also be used to help analyze the existing test suite. For
example, an analysis of test types can be performed to determine if a
variety of test types have been associated with each requirement. By
using a wide variety of types of tests, the tests are likely to be more
productive.
Examples of Test Types
Functional
Algorithmic
Positive
Negative
Usability
Boundary
Act Like a Customer
Startup
Shutdown
Configuration
Platform
Load/Stress
Security Performance
Data Flow
Safety-related
Compatibility
Documentation
Timing
Error Checking
Power Failure
Resources
Installation
Upgradeability
Volume Scalability
Throughput
In addition to looking for a variety of test types, there are specific
test strategies that are also important to identify. For example:
- Data Flow Testing
Data flow testing is a strategy based on known patterns of data usage
and data flow. Data flow tests are often derived from data flow
diagrams.
- Equivalence Classes
A group of tests can form an equivalence class if:
- they all test the same thing
- if one test passes, all tests will likely pass
- if one test fails, all tests will likely fail
Identifying potential equivalence classes allows redundant,
non-productive tests to be identified.
- Boundary Analysis
By identifying ranges and boundaries, tests can be created that are:
- well within boundary or range
- one less than boundary
- equal to boundary
- one greater than boundary
- well beyond boundary
Boundary analysis ensures that all boundary conditions are covered.
Again, this increases the likelihood that these tests are productive.
- Act Like a Customer Testing
Act Like a Customer (ALAC) testing is based on how users actually use
your product. These tests are critical in helping to uncover the kinds
of defects that users are likely to encounter. ALAC tests can be
easily created from Use Case/Workflow diagrams.
4 Scrub the Test Suite
The next step in the Testing Effectiveness process is to remove
redundant, non-productive tests from the test suite.
5 Identify Additional Training for Testers
Training for testers can help improve skills in areas such as:
- Test Planning
- Act Like a Customer Testing
- Equivalence Classes
- Data Flow Testing
- Boundary Analysis
- Requirements Management
- Test Automation
THE BOTTOM LINE
So, while it may not be practical to measure Testing ROI, improving
testing effectiveness will certainly improve the Testing ROI whether we
can measure it or not.
Till next time...
출처 : http://www.swqual.com
이올린에 북마크하기
