SE401 · Introduction · Annotated SlidesIntroduction to Software Testing
Original
Overview

Introduction to Software Testing

Why testing is necessary (famous failures, the cost of poor quality), errors vs. defects vs. failures, what testing is and its objectives, test concepts (test case, oracle, scaffolding, fixture, suite, adequacy), the seven testing principles, the fundamental test process, the psychology of testing and independence, and the limits of testing.

How to read this page

Every original slide is shown unchanged. Yellow marks on a slide point to a numbered note: hover or tap a mark to jump to its note, and hover a note to see its marks. Questions tagged Past exam appeared on a real paper; Prediction questions are likely angles, not confirmed. Pick an answer to each question: a wrong pick tells you why it is wrong, so you can try again.

All 94 slides
  1. 1 Midterm
  2. 2 Software is Everywhere
  3. 3 • Why is testing necessary? • What is testing?
  4. 4 Slide 4
  5. 5 Why Software Testing is Important?
  6. 6 Why Software Testing is Important?
  7. 7 Why Software Testing is Important?
  8. 8 Why Software Testing is Important?
  9. 9 • The Cost of Poor Software Quality in the US 2020 Report by CISQ Consortium for Information & Software Quality:
  10. 10 A Case Study: Ariane 5 Launch Vehicle
  11. 11 • T0 - T0 + 36s : normal • Within SRI 2:
  12. 12 • Not a programming error • unprotected conversion = design decision (~1980)
  13. 13 • Test! Test! Test! • Test! Even when the code is reused.
  14. 14 Why Software Testing is Important?
  15. 15 Congratulations!
  16. 16 • Testing is expensive.
  17. 17 testing fixing
  18. 18 • Testing adds to Cost of Conformance
  19. 19 Software systems context
  20. 20 • Human Error
  21. 21 • Both causes of errors produce defects( = faults, bugs) in the code. • Defects, if executed, may result in failures of the SW system (the
  22. 22 Four typical scenarios
  23. 23 Cost to repair
  24. 24 • Testing has an important role in all stages of a SW product’s life cycle:
  25. 25 • To reduce the risk of problems occurring during operation • To check if the SW system meets:
  26. 26 Testing …
  27. 27 Slide 27
  28. 28 • It is a systematic process used to identify the correctness , completeness , and quality of developed software. It includes a set of
  29. 29 • The process of testing all SW life-cycle activities:
  30. 30 • Depending on the objectives of the test process, testing can be focused on
  31. 31 • Finding defects
  32. 32 Slide 32
  33. 33 Myth #1 in Software Testing
  34. 34 Myth #2 in Software Testing
  35. 35 Q: How challenging is software testing? A: Testing is easier than design and
  36. 36 Q: How challenging is software testing?
  37. 37 Slide 37
  38. 38 • The program reads in 3 integer values that represent the lengths of the sides of a triangle.
  39. 39 Slide 39
  40. 40 Write a set of test cases that you feel would adequately test this program.
  41. 41 Test
  42. 42 Slide 42
  43. 43 The Very First Software Bug
  44. 44 • Failures are
  45. 45 • Defects are
  46. 46 • Errors are
  47. 47 • A human being makes an error ( mistake )
  48. 48 • An example : For any integer n, square (n) = n*n.
  49. 49 • An example : For any integer n, square (n) = n*n.
  50. 50 • Test Case Verdicts
  51. 51 Testing: Concepts
  52. 52 • Test
  53. 53 oracle
  54. 54 • Test Scaffolding
  55. 55 fixture: a fixed state of the software under test used as a baseline for
  56. 56 suite
  57. 57 • Test
  58. 58 • Test adequacy
  59. 59 • Test
  60. 60 Slide 60
  61. 61 • The purpose of testing
  62. 62 • Software testing is a planned process that is used to identify the correctness, completeness, security and quality of
  63. 63 • Testing is necessary, but not sufficient for quality assurance
  64. 64 Software Testing vs. Quality Assurance (QA)
  65. 65 Slide 65
  66. 66 The test principles
  67. 67 • Testing can show that defects are present, but cannot prove that there are no defects.
  68. 68 • Testing everything(all combinations of input and preconditions) is not feasible except for specific cases.
  69. 69 • Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined
  70. 70 • A small number of modules contains most of the defects discovered during pre release testing.
  71. 71 • If the same tests are repeated over and over again, the same set of test cases will no longer find any new bugs.
  72. 72 • Testing is done differently in different contexts.
  73. 73 • Finding and fixing defects does not help if the software system does not fulfill users' needs and expectations .
  74. 74 Slide 74
  75. 75 • Test case
  76. 76 • Test plan
  77. 77 • Test planning • Test monitoring and control
  78. 78 • Who, what , why, when and where
  79. 79 • Review test basis:
  80. 80 • Implement :
  81. 81 • Evaluate :
  82. 82 Slide 82
  83. 83 • Curiosity • Professional pessimism
  84. 84 • A certain degree of independence is often more effective at finding defects and failures.
  85. 85 • Independence levels:
  86. 86 • Be clear and objective
  87. 87 Slide 87
  88. 88 • Both, testing and verification attempts to exhibit new failures • Debugging is a systematic process that finds and eliminates the defect
  89. 89 • Myth: “Principles are just for reference. I will not use them in practice .”
  90. 90 • Can we exhaustively test a
  91. 91 “ Program testing can be used to show the presence of bugs, but never their absence. ”
  92. 92 How much testing is enough?
  93. 93 • One of the most difficult problems in testing is not knowing when to stop.
  94. 94 • Take into account
Slide 1 / 94

Midterm

Slide 1: Midterm
Slide 2 / 94

Software is Everywhere

Slide 2: Software is Everywhere
Slide 3 / 94

• Why is testing necessary? • What is testing?

Slide 3: • Why is testing necessary? • What is testing?
Slide 4 / 94

Slide 4

Slide 4: Slide 4

Notes

  1. Testing reduces the risk of failures in operation (Ariane 5, Therac-25, failed satellites, bank errors): poor quality is very expensive.

Slide 5 / 94

Why Software Testing is Important?

Slide 5: Why Software Testing is Important?
Slide 6 / 94

Why Software Testing is Important?

Slide 6: Why Software Testing is Important?
Slide 7 / 94

Why Software Testing is Important?

Slide 7: Why Software Testing is Important?
Slide 8 / 94

Why Software Testing is Important?

Slide 8: Why Software Testing is Important?
Slide 9 / 94

• The Cost of Poor Software Quality in the US 2020 Report by CISQ Consortium for Information & Software Quality:

Slide 9: • The Cost of Poor Software Quality in the US 2020 Report by CISQ Consortium for Information & Software Quality:
Slide 10 / 94

A Case Study: Ariane 5 Launch Vehicle

Slide 10: A Case Study: Ariane 5 Launch Vehicle
Slide 11 / 94

• T0 - T0 + 36s : normal • Within SRI 2:

Slide 11: • T0 - T0 + 36s : normal • Within SRI 2:

Notes

  1. Ariane 5 (1996): reused Ariane 4 code overflowed converting a 64-bit float to a 16-bit integer: not a coding slip, an untested assumption.

Slide 12 / 94

• Not a programming error • unprotected conversion = design decision (~1980)

Slide 12: • Not a programming error • unprotected conversion = design decision (~1980)
Slide 13 / 94

• Test! Test! Test! • Test! Even when the code is reused.

Slide 13: • Test! Test! Test! • Test! Even when the code is reused.
Slide 14 / 94

Why Software Testing is Important?

Slide 14: Why Software Testing is Important?
Slide 15 / 94

Congratulations!

Slide 15: Congratulations!
Slide 16 / 94

• Testing is expensive.

Slide 16: • Testing is expensive.
Slide 17 / 94

testing fixing

Slide 17: testing fixing
Slide 18 / 94

• Testing adds to Cost of Conformance

Slide 18: • Testing adds to Cost of Conformance
Slide 19 / 94

Software systems context

Slide 19: Software systems context
Slide 20 / 94

• Human Error

Slide 20: • Human Error
Slide 21 / 94

• Both causes of errors produce defects( = faults, bugs) in the code. • Defects, if executed, may result in failures of the SW system (the

Slide 21: • Both causes of errors produce defects( = faults, bugs) in the code. • Defects, if executed, may result in failures of the SW system (the
Slide 22 / 94

Four typical scenarios

Slide 22: Four typical scenarios
Slide 23 / 94

Cost to repair

Slide 23: Cost to repair
Slide 24 / 94

• Testing has an important role in all stages of a SW product’s life cycle:

Slide 24: • Testing has an important role in all stages of a SW product’s life cycle:
Slide 25 / 94

• To reduce the risk of problems occurring during operation • To check if the SW system meets:

Slide 25: • To reduce the risk of problems occurring during operation • To check if the SW system meets:
Slide 26 / 94

Testing …

Slide 26: Testing …
Slide 27 / 94

Slide 27

Slide 27: Slide 27
Slide 28 / 94

• It is a systematic process used to identify the correctness , completeness , and quality of developed software. It includes a set of

Slide 28: • It is a systematic process used to identify the correctness , completeness , and quality of developed software. It includes a set of
Slide 29 / 94

• The process of testing all SW life-cycle activities:

Slide 29: • The process of testing all SW life-cycle activities:
Slide 30 / 94

• Depending on the objectives of the test process, testing can be focused on

Slide 30: • Depending on the objectives of the test process, testing can be focused on
Slide 31 / 94

• Finding defects

Slide 31: • Finding defects
Slide 32 / 94

Slide 32

Slide 32: Slide 32
Slide 33 / 94

Myth #1 in Software Testing

Slide 33: Myth #1 in Software Testing
Slide 34 / 94

Myth #2 in Software Testing

Slide 34: Myth #2 in Software Testing
Slide 35 / 94

Q: How challenging is software testing? A: Testing is easier than design and

Slide 35: Q: How challenging is software testing? A: Testing is easier than design and
Slide 36 / 94

Q: How challenging is software testing?

Slide 36: Q: How challenging is software testing?
Slide 37 / 94

Slide 37

Slide 37: Slide 37
Slide 38 / 94

• The program reads in 3 integer values that represent the lengths of the sides of a triangle.

Slide 38: • The program reads in 3 integer values that represent the lengths of the sides of a triangle.
Slide 39 / 94

Slide 39

Slide 39: Slide 39
Slide 40 / 94

Write a set of test cases that you feel would adequately test this program.

Slide 40: Write a set of test cases that you feel would adequately test this program.
Slide 41 / 94

Test

Slide 41: Test
Slide 42 / 94

Slide 42

Slide 42: Slide 42
Slide 43 / 94

The Very First Software Bug

Slide 43: The Very First Software Bug
Slide 44 / 94

• Failures are

Slide 44: • Failures are

Notes

  1. A human makes an ERROR (mistake) → it leaves a DEFECT (fault, bug) in the code → running it may cause a FAILURE (wrong observable behaviour).

  2. Tip A human makes an ERROR (mistake) → it leaves a DEFECT (fault, bug) in the code → running it may cause a FAILURE (wrong observable behaviour).

Exam angle

PredictionQ1

A programmer types < instead of <=. The wrong operator in the code is a:

PredictionQ2

Every defect causes a failure.

Slide 45 / 94

• Defects are

Slide 45: • Defects are
Slide 46 / 94

• Errors are

Slide 46: • Errors are
Slide 47 / 94

• A human being makes an error ( mistake )

Slide 47: • A human being makes an error ( mistake )
Slide 48 / 94

• An example : For any integer n, square (n) = n*n.

Slide 48: • An example : For any integer n, square (n) = n*n.
Slide 49 / 94

• An example : For any integer n, square (n) = n*n.

Slide 49: • An example : For any integer n, square (n) = n*n.
Slide 50 / 94

• Test Case Verdicts

Slide 50: • Test Case Verdicts
Slide 51 / 94

Testing: Concepts

Slide 51: Testing: Concepts
Slide 52 / 94

• Test

Slide 52: • Test
Slide 53 / 94

oracle

Slide 53: oracle

Notes

  1. Test oracle: the mechanism that decides whether an output is correct (expected results, a reference implementation, assertions).

Exam angle

PredictionQ1

The source that tells you the expected result of a test is the:

Slide 54 / 94

• Test Scaffolding

Slide 54: • Test Scaffolding
Slide 55 / 94

fixture: a fixed state of the software under test used as a baseline for

Slide 55: fixture: a fixed state of the software under test used as a baseline for

Notes

  1. Fixture: the fixed state the tests run against. Suite: a collection of test cases. Scaffolding: drivers, stubs, harnesses.

Slide 56 / 94

suite

Slide 56: suite
Slide 57 / 94

• Test

Slide 57: • Test
Slide 58 / 94

• Test adequacy

Slide 58: • Test adequacy
Slide 59 / 94

• Test

Slide 59: • Test
Slide 60 / 94

Slide 60

Slide 60: Slide 60
Slide 61 / 94

• The purpose of testing

Slide 61: • The purpose of testing
Slide 62 / 94

• Software testing is a planned process that is used to identify the correctness, completeness, security and quality of

Slide 62: • Software testing is a planned process that is used to identify the correctness, completeness, security and quality of
Slide 63 / 94

• Testing is necessary, but not sufficient for quality assurance

Slide 63: • Testing is necessary, but not sufficient for quality assurance
Slide 64 / 94

Software Testing vs. Quality Assurance (QA)

Slide 64: Software Testing vs. Quality Assurance (QA)
Slide 65 / 94

Slide 65

Slide 65: Slide 65

Notes

  1. Seven principles: testing shows presence of defects (not absence) · exhaustive testing is impossible · early testing · defect clustering · pesticide paradox · testing is context dependent · absence-of-errors fallacy.

  2. Tip Seven principles: testing shows presence of defects (not absence) · exhaustive testing is impossible · early testing · defect clustering · pesticide paradox · testing is context dependent · absence-of-errors fallacy.

Slide 66 / 94

The test principles

Slide 66: The test principles
Slide 67 / 94

• Testing can show that defects are present, but cannot prove that there are no defects.

Slide 67: • Testing can show that defects are present, but cannot prove that there are no defects.
Slide 68 / 94

• Testing everything(all combinations of input and preconditions) is not feasible except for specific cases.

Slide 68: • Testing everything(all combinations of input and preconditions) is not feasible except for specific cases.
Slide 69 / 94

• Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined

Slide 69: • Testing activities should start as early as possible in the software or system development life cycle and should be focused on defined
Slide 70 / 94

• A small number of modules contains most of the defects discovered during pre release testing.

Slide 70: • A small number of modules contains most of the defects discovered during pre release testing.
Slide 71 / 94

• If the same tests are repeated over and over again, the same set of test cases will no longer find any new bugs.

Slide 71: • If the same tests are repeated over and over again, the same set of test cases will no longer find any new bugs.

Exam angle

PredictionQ1

Running the same tests over and over stops finding new defects. This is the:

Slide 72 / 94

• Testing is done differently in different contexts.

Slide 72: • Testing is done differently in different contexts.
Slide 73 / 94

• Finding and fixing defects does not help if the software system does not fulfill users' needs and expectations .

Slide 73: • Finding and fixing defects does not help if the software system does not fulfill users' needs and expectations .

Notes

  1. Absence-of-errors fallacy: a bug-free system that doesn’t meet users’ needs is still a failure.

Slide 74 / 94

Slide 74

Slide 74: Slide 74

Notes

  1. Fundamental test process: planning and control → analysis and design → implementation and execution → evaluating exit criteria and reporting → test closure.

Slide 75 / 94

• Test case

Slide 75: • Test case
Slide 76 / 94

• Test plan

Slide 76: • Test plan
Slide 77 / 94

• Test planning • Test monitoring and control

Slide 77: • Test planning • Test monitoring and control
Slide 78 / 94

• Who, what , why, when and where

Slide 78: • Who, what , why, when and where
Slide 79 / 94

• Review test basis:

Slide 79: • Review test basis:
Slide 80 / 94

• Implement :

Slide 80: • Implement :
Slide 81 / 94

• Evaluate :

Slide 81: • Evaluate :
Slide 82 / 94

Slide 82

Slide 82: Slide 82

Notes

  1. Psychology: testers need curiosity, professional pessimism, attention to detail, good communication; report defects objectively.

Slide 83 / 94

• Curiosity • Professional pessimism

Slide 83: • Curiosity • Professional pessimism
Slide 84 / 94

• A certain degree of independence is often more effective at finding defects and failures.

Slide 84: • A certain degree of independence is often more effective at finding defects and failures.

Notes

  1. Independence levels: the author tests their own code → another developer → an independent test team → outside organization. More independence finds different defects.

Slide 85 / 94

• Independence levels:

Slide 85: • Independence levels:
Slide 86 / 94

• Be clear and objective

Slide 86: • Be clear and objective
Slide 87 / 94

Slide 87

Slide 87: Slide 87
Slide 88 / 94

• Both, testing and verification attempts to exhibit new failures • Debugging is a systematic process that finds and eliminates the defect

Slide 88: • Both, testing and verification attempts to exhibit new failures • Debugging is a systematic process that finds and eliminates the defect
Slide 89 / 94

• Myth: “Principles are just for reference. I will not use them in practice .”

Slide 89: • Myth: “Principles are just for reference. I will not use them in practice .”
Slide 90 / 94

• Can we exhaustively test a

Slide 90: • Can we exhaustively test a
Slide 91 / 94

“ Program testing can be used to show the presence of bugs, but never their absence. ”

Slide 91: “ Program testing can be used to show the presence of bugs, but never their absence. ”
Slide 92 / 94

How much testing is enough?

Slide 92: How much testing is enough?
Slide 93 / 94

• One of the most difficult problems in testing is not knowing when to stop.

Slide 93: • One of the most difficult problems in testing is not knowing when to stop.
Slide 94 / 94

• Take into account

Slide 94: • Take into account