Show understanding of ways of exposing and avoiding faults in programs

Program Testing and Maintenance

Learning Objective

Show understanding of ways of exposing and avoiding faults in programmes (Cambridge syllabus 12.3).

Key Terminology (Cambridge wording)

• Fault (defect): A flaw in the design or source code that may cause an error when the programme is executed.
• Error: A human mistake (e.g., a logic mistake in an algorithm) that introduces a fault.
• Failure: The observable incorrect behaviour of a programme caused by a fault that is executed.

Types of Faults – AO Mapping

Fault‑identification checklist (mirrors syllabus wording)

1. Identify the observed problem (failure).
2. Locate the part of the programme where the problem occurs (e.g., line number, module).
3. Classify the fault (syntax, logic, run‑time).
4. Explain why the fault causes the failure (analysis).
5. Propose a concrete correction and, where appropriate, a test to confirm the fix.

Testing Overview

Static vs Dynamic Testing

• Static testing: Examine artefacts (source code, design documents, test specifications) without executing the programme.

  • Techniques: code reviews, walkthroughs, inspections, static‑analysis tools.
  • Primarily AO1 – recognise potential faults.

• Dynamic testing: Run the programme with selected inputs and observe the output or behaviour.

  • Can be performed manually or with automated test harnesses.
  • Addresses AO2 (analyse) and AO3 (evaluate/correct).

Automated vs Manual Testing (syllabus note)

• Automated testing: Test scripts or frameworks execute test cases repeatedly (e.g., JUnit, pytest). Benefits: repeatability, regression testing, coverage measurement.
• Manual testing: Human tester selects inputs, observes results, and records outcomes. Useful for exploratory, usability or ad‑hoc testing.

White‑box, Black‑box and Grey‑box Testing

Test Levels – Matrix with AO Mapping

Designing Effective Test Cases

1. Equivalence Partitioning: Divide the input domain into classes that are expected to behave alike; select one representative value from each class.
2. Boundary Value Analysis: Test values at, just inside and just outside the limits of each equivalence class.
3. Decision‑Table Testing – worked example:

   Condition	Username valid?	Password valid?
   C1	Yes	Yes
   C2	Yes	No
   C3	No	Yes
   C4	No	No

   Actions derived from the table:

   • C1 → grant access
   • C2 or C3 → reject – “invalid credentials”
   • C4 → reject – “username and password required”

   Four test cases cover all possible condition combinations.

4. State‑Transition Testing: Model the system as a finite‑state machine and create test cases that traverse every transition (e.g., login → logged‑in → logout → logged‑out).
5. Use‑Case Testing: Derive test cases from typical user scenarios described in the use‑case diagram or narrative.

Mini‑Exercise – Applying EP and BVA

Given the pseudo‑code below, produce test cases using equivalence partitioning and boundary analysis.

function grade(score):
if score < 0 or score > 100:
return "invalid"
else if score >= 70:
return "A"
else if score >= 50:
return "B"
else:
return "C"

Expected answer:

• Equivalence classes: invalid‑low (score < 0), invalid‑high (score > 100), A (70‑100), B (50‑69), C (0‑49).
• Boundary values: 0, 49, 50, 69, 70, 100, 101 (and –1 for the low side).

Test Strategy & Test Plan – What to Include

A concise test plan demonstrates that the student can select appropriate test data, describe the testing approach and anticipate the evidence needed. The plan should contain:

• Scope: Which modules, features or requirements are covered.
• Objectives: What faults the plan aims to expose (e.g., boundary errors, interface mismatches).
• Test levels & techniques: Specify unit, integration, system and/or acceptance testing and whether white‑box, black‑box or grey‑box methods are used.
• Test data selection: Explain the use of equivalence partitions, boundary values, decision tables, etc.
• Test environment: Hardware, OS, compiler/interpreter, any required libraries or tools (including automated test frameworks).
• Pass/fail criteria: Expected output or behaviour for each test case.
• Schedule & resources: Approximate time, personnel and tools required.
• Risk & mitigation: Identify high‑risk areas and additional testing that will be performed.

Metrics for Test Coverage

Coverage quantifies how much of the programme has been exercised by the test suite.

Statement coverage:

\$C_{stmt} = \frac{\text{Number of statements executed}}{\text{Total number of statements}} \times 100\%\$

Branch (decision) coverage:

\$C_{branch} = \frac{\text{Number of decision outcomes exercised}}{\text{Total number of decision outcomes}} \times 100\%\$

Sample Calculation

Consider the fragment:

1  if (x > 0)          // decision D1
2      y = 10;
3  else
4      y = -10;
5  if (y > 5)          // decision D2
6      z = y * 2;
7  else
8      z = y / 2;

Test suite:

• Test A: x = 1 → executes 1‑2‑5‑6 (D1‑true, D2‑true).
• Test B: x = -1 → executes 1‑3‑4‑5‑7‑8 (D1‑false, D2‑false).

Results:

• All 8 statements executed → 100 % statement coverage.
• Both outcomes of D1 and D2 exercised → 100 % branch coverage.

Exposing Faults

• Code reviews & walkthroughs: Peer inspection of design and source code to spot faults early (static testing).
• Static analysis tools: Automated scanners that flag common defects (unused variables, possible null dereference, style violations).
• Debugging: Use break‑points, watch variables and step‑through execution to locate the source of a failure.
• Profiling: Measure runtime characteristics (time, memory) to uncover performance‑related faults.
• Automated test suites & regression testing: Re‑run the full set of tests after any change to catch newly introduced faults.

Avoiding Faults – “Why It Matters”

• Good design practices: Clear modular structure and well‑defined interfaces limit the impact of a change, reducing logic faults.
• Structured programming: Using only sequence, selection and iteration constructs keeps code readable and easier to reason about.
• Coding standards: Consistent naming, indentation and commenting avoid misunderstandings that can lead to syntax or logic faults.
• Defensive programming: Validate all inputs and handle exceptional conditions to prevent run‑time faults such as crashes or security breaches.
• Code reuse & libraries: Re‑using well‑tested components lowers the probability of introducing new faults.
• Comprehensive documentation: Up‑to‑date design and API documents ensure developers understand intended behaviour.
• Version control: Tracks changes, enables rollback and supports collaborative development, helping to locate the introduction of a fault quickly.
• Test‑driven development (TDD): Writing tests before code clarifies requirements, forces small, testable units and catches faults at the earliest stage.

Maintenance Types – Illustrated Case Study

Scenario: A simple student‑record system has been in use for three years.

1. Corrective maintenance – A bug: entering a negative grade crashes the programme.

   • Fix: add a check if grade < 0 then reject.
   • Testing: re‑test the grade‑entry module and run the full regression suite.

2. Adaptive maintenance – The school upgrades to a new operating system.

   • Fix: re‑compile with the new compiler, replace deprecated library calls.
   • Testing: integration testing of the file‑I/O module on the new OS.

3. Perfective maintenance – Users request faster search of student records.

   • Fix: replace linear search with a binary‑search algorithm.
   • Testing: add performance test cases; verify correctness with existing unit tests.

4. Preventive maintenance – Code review reveals duplicated validation code.

   • Fix: refactor validation into a single helper routine and update documentation.
   • Testing: run the full regression suite; update the design diagram.

Testing During Maintenance

• Regression testing: Re‑execute the full set of existing test cases after any change to ensure no previously‑working functionality has broken.
• Re‑testing: Run the specific test case that originally failed to confirm the fault is now fixed.
• Impact analysis: Identify which modules, functions or data structures may be affected by a change and concentrate testing effort on those areas.

Summary Checklist (Student Revision)

1. Identify potential fault sources (design, code, requirements) and use the fault‑identification checklist.
2. Choose the appropriate test level (unit → integration → system → acceptance) and technique (static, dynamic, white‑box, black‑box, grey‑box).
3. Design test cases using equivalence partitioning, boundary analysis, decision tables, state‑transition or use‑case techniques.
4. Calculate statement and branch coverage; aim for high percentages and record the results.
5. Carry out static testing (reviews, analysis) before dynamic testing.
6. Apply defensive programming, coding standards, structured design, version control and, where appropriate, TDD to avoid faults.
7. Maintain up‑to‑date documentation and use version control for traceability.
8. During maintenance, perform impact analysis, regression testing and update test suites as required.