200 test cases, 160 executed, 120 passed. The pass rate of executed tests is:
- 120 / 160.
Testing Metrics and Tools
Why measure testing, base vs. calculated metrics (test-case productivity, execution and pass rates, defect density, defect leakage, defect removal efficiency, coverage), test tool classes (management, static, design, execution, performance, monitoring), benefits and risks of tools, and interview questions on QA vs. QC and verification vs. validation.
How to read this page
Every original slide is shown unchanged. Yellow marks on a slide point to a numbered note: hover or tap a mark to jump to its note, and hover a note to see its marks. Questions tagged Past exam appeared on a real paper; Prediction questions are likely angles, not confirmed. Pick an answer to each question: a wrong pick tells you why it is wrong, so you can try again.
All 97 slides
- 1 Slide 1
- 2 • Testing Metrics • Testing Tools
- 3 • Metrics can be defined as “STANDARDS OF MEASUREMENT” • Metric is a unit used for describing or measuring an attribute
- 4 • "We cannot improve what we cannot measure" and Test Metrics helps us to do exactly the same.
- 5 • Software testing metrics or software test measurement is the quantitative indication of extent, capacity, dimension, amount or size of
- 6 • You cannot improve what you cannot measure
- 7 • Take decision for next phase of activities • Evidence of the claim or prediction
- 8 test metrics is to use the data for
- 9 Types of Test Metrics
- 10 • Identification of correct testing metrics is very important. Few things need to be considered before identifying the test metrics
- 11 • Manual test metrics are classified into two classes
- 12 • Base metrics is the raw data collected by Test Analyst during the test case development and execution (# of test cases executed, # of test
- 13 • Major Base Metrics:
- 14 • Major Calculated Metrics:
- 15 • Major Calculated Metrics:
- 16 • Major Calculated Metrics:
- 17 • Major Calculated Metrics:
- 18 • Major Calculated Metrics:
- 19 • Major Calculated Metrics:
- 20 • Major Calculated Metrics:
- 21 • Major Calculated Metrics:
- 22 • Major Calculated Metrics:
- 23 • Major Calculated Metrics:
- 24 • Major Calculated Metrics:
- 25 • Major Calculated Metrics:
- 26 • Major Calculated Metrics:
- 27 • Major Calculated Metrics:
- 28 • Major Calculated Metrics:
- 29 • The total amount of time spent actually testing the product measured in test hours
- 30 • Code Coverage (How much of the code is being exercised?)
- 31 • Defect removal percentage
- 32 • Fundamental test metrics are a combination of Base Metrics that can then be used to produce Calculated Metrics .
- 33 • Major Calculated Software Test
- 34 Slide 34
- 35 Slide 35
- 36 • Test tools can be used for one or more activities that support testing:
- 37 • Tools can have one or more purposes, depending on the context:
- 38 • Tools are classified according to the testing activities that they support.
- 39 • Characteristics
- 40 • Test management tools and application lifecycle management tool (ALM)
- 41 • Tools for static testing
- 42 • Review process tools
- 43 • Static analysis tools (D)
- 44 • Static analysis tools (D)
- 45 Static analysis tools (D)
- 46 • Static analysis tool example: Source Monitor
- 47 • Static analysis tool example: Source Monitor
- 48 • JDepend
- 49 • Modeling tools (D)
- 50 • Modeling tool example: StarUML
- 51 • The major benefit of static testing tools and modeling tools is the cost effectiveness of finding more defects at an earlier time in the
- 52 • Test design tools • Model-based testing tools
- 53 • Test design tools
- 54 • Model-based testing
- 55 • TDD -Test driven development tool (D)
- 56 • Test execution tools • Coverage tools
- 57 • Test execution tools
- 58 • Test harnesses (D) and Unit test framework tools (D)
- 59 • Drivers
- 60 • Test harnesses (D) and Unit test framework tools (D)
- 61 • Characteristics of test harness and unit test framework tools
- 62 • Test comparators
- 63 • Test data for performance testing
- 64 • Tests should reflect realistic (correct) scenarios
- 65 • Common features of test data preparation tools
- 66 • Monitoring tools identifying problems and sending an alert message to the administrator (e.g. network administrator)
- 67 Monitoring tools
- 68 • Features or characteristics of dynamic analysis tools include support for
- 69 Dynamic analysis tools (D)
- 70 • There are tools specialized for use in a particular type of application
- 71 • There are tools specialized for use in a particular type of application. • Example:
- 72 • Testers may use:
- 73 • Simply purchasing or leasing a tool does not guarantee success with that tool!
- 74 • Reduce repetitive work (running regression tests, re-entring the same test data, etc.)
- 75 • Greater consistency and repeatability
- 76 • Objective assessment
- 77 • Ease of access to information about the tests or test effort
- 78 • Unrealistic expectations for the tool (functionality and ease of use) • Underestimating time, cost, effort for the introduction of a tool (training,
- 79 • Tools are not magic! • They can do very well what they have been designed to do, but they
- 80 • This type of tool often requires significant effort in order to achieve significant benefits.
- 81 • The design of the load to be generated by the tool • Timing aspects probe effect
- 82 • There is a risk that the changes to make old code to conform to new standard will introduce an unexpected side effect
- 83 • They need to interface with other tools or spreadsheets in order to produce information in the best format for the current needs of the
- 84 Slide 84
- 85 • When can we stop testing?
- 86 • What is difference between QA, QC and Software Testing?
- 87 • Quality Assurance (QA) : QA refers to the planned and systematic way of monitoring the quality of process which is followed to produce a quality product. QA
- 88 What is verification and validation? • Verification: process of evaluating work-products of a development
- 89 Explain Branch Coverage and Decision Coverage. • Branch Coverage is testing performed in order to ensure that every
- 90 • The term "static analysis" is conflated, but here we use it to mean a collection of algorithms and techniques used to analyze source code in
- 91 Questions:
- 92 • Testing “Phases” • Static vs. Dynamic Testing
- 93 • Definition - What does Quality Assurance (QA)
- 94 • Software quality assurance (SQA) is a process that ensures that developed
- 95 • The area of Software Quality Assurance can be broken down into a number of smaller areas such as
- 96 don’t build it
- 97 • Recognizing the ACM and IEEE code of ethics for engineers: • PUBLIC - Software testers shall act consistently with the public interest. • CLIENT AND EMPLOYER - Software testers shall act in a manner that is in the best interests of their client and
Slide 1
• Testing Metrics • Testing Tools
• Metrics can be defined as “STANDARDS OF MEASUREMENT” • Metric is a unit used for describing or measuring an attribute
Notes
"You cannot improve what you cannot measure": metrics show progress, quality and where to act.
• "We cannot improve what we cannot measure" and Test Metrics helps us to do exactly the same.
• Software testing metrics or software test measurement is the quantitative indication of extent, capacity, dimension, amount or size of
• You cannot improve what you cannot measure
• Take decision for next phase of activities • Evidence of the claim or prediction
test metrics is to use the data for
Types of Test Metrics
• Identification of correct testing metrics is very important. Few things need to be considered before identifying the test metrics
• Manual test metrics are classified into two classes
• Base metrics is the raw data collected by Test Analyst during the test case development and execution (# of test cases executed, # of test
Notes
Base metrics: raw counts (test cases written, executed, passed, failed, blocked; defects found). Calculated metrics are derived from them.
• Major Base Metrics:
• Major Calculated Metrics:
Notes
Examples: % test cases executed = executed / total × 100; pass rate = passed / executed × 100; defect density = defects / size (KLOC).
Exam angle
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• Major Calculated Metrics:
• The total amount of time spent actually testing the product measured in test hours
• Code Coverage (How much of the code is being exercised?)
• Defect removal percentage
Notes
Defect removal efficiency (DRE) = defects found before release / (before + after release) × 100.
Exam angle
90 defects found in testing, 10 found by customers after release. DRE =
- 90 / 100.
• Fundamental test metrics are a combination of Base Metrics that can then be used to produce Calculated Metrics .
• Major Calculated Software Test
Slide 34
Slide 35
• Test tools can be used for one or more activities that support testing:
Notes
Tools support test management, static testing, test design, execution and logging, performance and monitoring.
• Tools can have one or more purposes, depending on the context:
• Tools are classified according to the testing activities that they support.
• Characteristics
• Test management tools and application lifecycle management tool (ALM)
• Tools for static testing
• Review process tools
• Static analysis tools (D)
Notes
Static analysis tools (SpotBugs, checkstyle, JDepend) find defects in code without running it.
• Static analysis tools (D)
Static analysis tools (D)
• Static analysis tool example: Source Monitor
• Static analysis tool example: Source Monitor
• JDepend
• Modeling tools (D)
• Modeling tool example: StarUML
• The major benefit of static testing tools and modeling tools is the cost effectiveness of finding more defects at an earlier time in the
• Test design tools • Model-based testing tools
• Test design tools
• Model-based testing
• TDD -Test driven development tool (D)
• Test execution tools • Coverage tools
• Test execution tools
Notes
Test execution tools run scripted tests; harnesses and unit frameworks provide drivers and stubs.
• Test harnesses (D) and Unit test framework tools (D)
• Drivers
• Test harnesses (D) and Unit test framework tools (D)
• Characteristics of test harness and unit test framework tools
• Test comparators
• Test data for performance testing
• Tests should reflect realistic (correct) scenarios
• Common features of test data preparation tools
• Monitoring tools identifying problems and sending an alert message to the administrator (e.g. network administrator)
Monitoring tools
• Features or characteristics of dynamic analysis tools include support for
Dynamic analysis tools (D)
• There are tools specialized for use in a particular type of application
• There are tools specialized for use in a particular type of application. • Example:
• Testers may use:
• Simply purchasing or leasing a tool does not guarantee success with that tool!
• Reduce repetitive work (running regression tests, re-entring the same test data, etc.)
Notes
Benefits: less repetitive work, consistency, objective assessment. Risks: unrealistic expectations, underestimated effort, over-reliance on the tool.
• Greater consistency and repeatability
• Objective assessment
• Ease of access to information about the tests or test effort
• Unrealistic expectations for the tool (functionality and ease of use) • Underestimating time, cost, effort for the introduction of a tool (training,
• Tools are not magic! • They can do very well what they have been designed to do, but they
• This type of tool often requires significant effort in order to achieve significant benefits.
• The design of the load to be generated by the tool • Timing aspects probe effect
• There is a risk that the changes to make old code to conform to new standard will introduce an unexpected side effect
• They need to interface with other tools or spreadsheets in order to produce information in the best format for the current needs of the
Slide 84
• When can we stop testing?
Notes
Stop testing when exit criteria are met: coverage, defect rates, deadlines, budget, acceptable residual risk.
• What is difference between QA, QC and Software Testing?
• Quality Assurance (QA) : QA refers to the planned and systematic way of monitoring the quality of process which is followed to produce a quality product. QA
What is verification and validation? • Verification: process of evaluating work-products of a development
Explain Branch Coverage and Decision Coverage. • Branch Coverage is testing performed in order to ensure that every
• The term "static analysis" is conflated, but here we use it to mean a collection of algorithms and techniques used to analyze source code in
Questions:
• Testing “Phases” • Static vs. Dynamic Testing
• Definition - What does Quality Assurance (QA)
• Software quality assurance (SQA) is a process that ensures that developed
• The area of Software Quality Assurance can be broken down into a number of smaller areas such as
don’t build it
• Recognizing the ACM and IEEE code of ethics for engineers: • PUBLIC - Software testers shall act consistently with the public interest. • CLIENT AND EMPLOYER - Software testers shall act in a manner that is in the best interests of their client and
