Testing Maturity Model Integration
TMMi
Testing Maturity Model Integration
The software industry is still fighting with defects and cost of handling defects since the day first software developed. The cost of Testing and quality activities occupy 40-45% of the total budget. Even though such large sum of money is spent, still software produced with simple defects uncovered and identified at production stage. I see at high level there may be two reasons
Factor#1:
The Importance of process is felt most of the times after the failure
Factor #2:
Integrated complete life cycle of process is not followed even in matured organizations
TMMi , is a testing maturity assessment model in line with CMMi ( Capability Maturity Model Integration) . TMMi acts as a counterpart to CMMi in software engineering. TMMi intended to improve the way software tested and improve the testing process to make better software quality.
TMMi Maturity levels
Level 1: Initial
Level 2: Managed
2.1 Test policy and strategy
2.2 Test Planning
2.3 Test Monitoring and Control
2.4 Test Design and Execution
2.5 Test Environment
Level 3: Defined
3.1 Test Organization
3.2 Test Training program
3.3 Test Lifecycle and Integration
3.4 Non-Functional Testing
3.5 Peer reviews
Level 4: Measured
4.1 Test Measurement
4.2 Software Quality Evaluation
4.3 Advanced Peer reviews
Level 5: Optimizing
5.1 Defect Prevention
5.2 Test Process Optimization
5.3 Quality Control
Ref: http://www.tmmifoundation.org/downloads/tmmi/TMMi%20Framework.pdf
CMMI KPAs
22 KPAs are defined across all the 5 levels of CMMI as per CMMI version 1.2
Level 1: No KPAs defined at this level.
Level 2: 7 KPAs are defined at this Level
- Configuration Management (CM)
- Measurement and Analysis (MA)
- Project Monitoring and Control (PMC)
- Project Planning (PP)
- Process and Product Quality Assurance (PPQA)
- Requirements Management (REQM)
- Supplier Agreement Management (SAM)
Level 3: 11 KPAs are defined at this Level, plus KPAs defined at Level 2
- Decision Analysis and Resolution (DAR)
- Integrated Project Management (IPM)
- Organizational Process Definition (OPD)
- Organizational Process Focus (OPF)
- Organizational Training (OT)
- Product Integration (PI)
- Requirements Development (RD)
- Risk Management (RSKM)
- Technical Solution (TS)
- Validation (VL)
- Verification (VER)
Maturity Level 4: 2 KPAs at defined at this Level, plus KPAs defined at Levels 3 & 2
- Quantitative Project Management (QPM)
- Organizational Process Performance (OPP)
Maturity Level 5: 2 KPAs at defined at this Level, plus KPAs defined at Levels 4, 3 & 2
- Causal Analysis and Resolution (CAR)
- Organizational Innovation and Deployment (OID)
Testing Techniques
General Techniques of Testing:
There are two main methodologies of testing: white-box and black-box testing.
White box testing
Structural tests verify the structure of the software itself and require complete access to the object’s source code. This is known as ‘white box’ testing because you see into the internal workings of the code.
White-box tests make sure that the software structure itself contributes to proper and efficient program execution. Complicated loop structures, common data areas, 100,000 lines of spaghetti code and nests of its are evil. Well-designed control structures, sub-routines and reusable modular programs are good.
Many studies show that the single most effective defect reduction process is the classic structural test – the code inspection or walk-through. Code inspection is like proofreading – it can find the mistakes the author missed – the “typo’s” and logic errors that even the best programmers can produce. Debuggers are typical white-box tools.
White-box tasting strength is also its weakness. The code needs to be examined – by highly skilled technicians. That means that tools and skills are highly specialized to the particular language and environment. Also, large or distributed system execution goes beyond one program, so a correct procedure might call another program that provides bad data. In large systems, it is the execution path as defined by the program calls, their input and output and the structure of common files that is important. This gets into a hybrid kind of testing that is often employed in intermediate or integration stages of testing.
White box testing defined as “Planning and designing test documents based on internal code structure”
This technique of testing is also referred as
v Structural Testing technique
v Clear Box Testing technique
v Glass Box Testing technique
Black Box testing:
This is the technique which is used by independent testers to enhance the quality of application.
Functional tests examine the observable behavior of software as evidenced by its outputs without reference to internal functions. Hence ‘black box’ testing. If the program consistently provides the desired features with acceptable performance, then specific source code features are irrelevant. It’s a pragmatic and down-to-earth assessment of software.
Black box tests better address the modern programming paradigm. As object-oriented programming, automatic code generation and code re-use becomes more prevalent, analysis of source code itself becomes less important and functional tests become more important.
Black box tests also better attack the quality target. Since only the people paying for an application can determine if it meets their needs, it is an advantage to create the quality criteria from this point of view from the beginning.
Black box tests have a basis in the scientific method. Like the process of science, functional tests must have a hypothesis (your specifications), a defined method or procedure (your test), reproducible components (your test data), and a standard notation to record the results.
“Planning and designing test documents s with the knowledge of inputs, outputs and functional requirements of program. Without looking at contents or implementation”
You can re-run black box tests after a change to make sure the change only produced intended results with no inadvertent effects.
Gray Box testing:
This method of testing holds the above mentioned both techniques, this is done if this technique is recommended by the higher authorities of the product team. This will be recommended keeping many constraints in mind such as time, cost and schedule changes.
Static and Dynamic testing:
Any static application is fixed and does not give any response to request. Static is performed without executing the operational programs. The system which is responding to request requires testing techniques that simulate the dynamic environment in order to ascertain that the systems will function properly when they undergo dynamic testing. Generally Static tests are used in the requirements and design phase, and dynamic tests in the program test phase.
| Taxonomy | Black Box Testing | White Box Testing |
| Dynamic | v Random Testingv Domain Testing (Functionality Based) | v Computation testingv Domain testing Path-based
v Mutation analysis |
| Static | v Requirement-Specificationv Matching | v Code walkthroughsv Inspections Program
v Proving execution |
software testing metrics
Testing Metrics
A software metric is a measure of some property of a piece of software or its specifications.
Since quantitative methods have proved so powerful in the other sciences, computer science practitioners and theoreticians have worked hard to bring similar approaches to software development. Tom DeMarco stated, “You can’t control what you can’t measure.”
“The Product Quality Measures “
1. Customer satisfaction index
This index is surveyed before product delivery and after product delivery
(and on-going on a periodic basis, using standard questionnaires).The following are analyzed:
– Number of system enhancement requests per year
– Number of maintenance fix requests per year
– User friendliness: call volume to customer service hotline
– User friendliness: training time per new user
– Number of product recalls or fix releases (software vendors)
– Number of production re-runs (in-house information systems groups)
2. Delivered defect quantities
They are normalized per function point (or per LOC) at product delivery (first 3 months or first year of operation) or Ongoing (per year of operation) by level of severity, by category or cause, e.g.: requirements defect, design defect, code defect, documentation/on-line help defect, defect introduced by fixes, etc.
3. Responsiveness (turnaround time) to users
– Turnaround time for defect fixes, by level of severity
– Time for minor vs. major enhancements; actual vs. planned elapsed time
4. Product volatility
– Ratio of maintenance fixes (to repair the system & bring it into compliance with specifications), vs. enhancement requests (requests by users to enhance or change functionality)
5. Defect ratios
– Defects found after product delivery per function point.
– Defects found after product delivery per LOC
– Pre-delivery defects: annual post-delivery defects
– Defects per function point of the system modifications
6. Defect removal efficiency
– Number of post-release defects (found by clients in field operation), categorized by level of severity
– Ratio of defects found internally prior to release (via inspections and testing), as a percentage of all defects
– All defects include defects found internally plus externally (by customers) in the first year after product delivery
7. Complexity of delivered product
– McCabe’s cyclomatic complexity counts across the system
– Halstead’s measure
– Card’s design complexity measures
– Predicted defects and maintenance costs, based on complexity measures
8. Test coverage
– Breadth of functional coverage
– Percentage of paths, branches or conditions that were actually tested
– Percentage by criticality level: perceived level of risk of paths
– The ratio of the number of detected faults to the number of predicted faults.
9. Cost of defects
– Business losses per defect that occurs during operation
– Business interruption costs; costs of work-arounds
– Lost sales and lost goodwill
– Litigation costs resulting from defects
– Annual maintenance cost (per function point)
– Annual operating cost (per function point)
– Measurable damage to your boss’s career
10. Costs of quality activities
– Costs of reviews, inspections and preventive measures
– Costs of test planning and preparation
– Costs of test execution, defect tracking, version and change control
– Costs of diagnostics, debugging and fixing
– Costs of tools and tool support
– Costs of test case library maintenance
– Costs of testing & QA education associated with the product
– Costs of monitoring and oversight by the QA organization (if separate from the development and test organizations)
11. Re-work
- Re-work effort (hours, as a percentage of the original coding hours)
– Re-worked LOC (source lines of code, as a percentage of the total delivered LOC)
– Re-worked software components (as a percentage of the total delivered components)
12. Reliability
– Availability (percentage of time a system is available, versus the time the system is needed to be available)
– Mean time between failure (MTBF).
– Man time to repair (MTTR)
– Reliability ratio (MTBF / MTTR)
– Number of product recalls or fix releases
– Number of production re-runs as a ratio of production runs
Metrics for Evaluating Application System Testing:
Metric => Formula
Test Coverage = Number of units (KLOC/FP) tested / total size of the system. (LOC represents Lines of Code)
Number of tests per unit size = Number of test cases per KLOC/FP (LOC represents Lines of Code).
Acceptance criteria tested = Acceptance criteria tested / total acceptance criteria
Defects per size = Defects detected / system size
Test cost (in %) = Cost of testing / total cost *100
Cost to locate defect = Cost of testing / the number of defects located
Achieving Budget = Actual cost of testing / Budgeted cost of testing
Defects detected in testing = Defects detected in testing / total system defects
Defects detected in production = Defects detected in production/system size
Quality of Testing = No of defects found during Testing/(No of defects found during testing + No of acceptance defects found after delivery) *100
Effectiveness of testing to business = Loss due to problems / total resources processed by the system.
System complaints = Number of third party complaints / number of transactions processed
Scale of Ten = Assessment of testing by giving rating in scale of 1 to 10
Source Code Analysis = Number of source code statements changed / total number of tests.
Effort Productivity = Test Planning Productivity = No of Test cases designed / Actual
Effort for Design and Documentation
Test Execution Productivity = No of Test cycles executed / Actual
