Foundations

Overview

This first course is a basic introduction to black box testing. It presents basic terminology and considers:

• the mission of testing

• the oracle problem

• the measurement problem

• the impossibility of complete testing

More info on the Learning Objectives for Foundations of Software Testing are available on the BBST.info website.

Slides

• Foundations Slides - Latest

• Foundations Slides - Original

Videos

Lecture 1: Overview & Basic Definitions

This provides an overview of the online Black Box Software Testing courses and introduces some definitions commonly used in the testing field.

• Lecture 1A

• Lecture 1B

• Lecture 1C

Lecture 2: Strategy

This considers why testers test, what they are trying to learn, and how they can organize their work to achieve their mission.

• Lecture 2A
• Lecture 2B

Lecture 3: Oracles

This presents software oracles as heuristics that help testers make a judgment whether or not software passes the tests that are run.

• Lecture 3A
• Lecture 3B
• Lecture 3C

Lecture 4: Programming Fundamentals & Coverage

This presents information about basic data handling and storage to help testers think about the multi-dimensional problem of test coverage in more sophisticated ways.

• Lecture 4A
• Lecture 4B
• Lecture 4C
• Lecture 4D

Lecture 5: The Impossibility of Complete Testing

This explores the complexity of determining when testing is finished and how the goal of complete testing is unattainable.

• Lecture 5A
• Lecture 5B
• Lecture 5C

Lecture 6: Introduction to Measurement

This addresses the challenges of measurement in software testing.

• Lecture 6A
• Lecture 6B
• Lecture 6C

Readings

Required Readings

• Michael Bolton: Testing Without a Map
• Douglas Hoffman: Exhausting your test options
• Cem Kaner: The impossibility of complete testing
• Cem Kaner: Software negligence and testing coverage
• Cem Kaner, Elisabeth Hendrickson & Jennifer Smith-Brock: Managing the proportion of testers to (other) developers
• Brian Marick: How to misuse code coverage

Recommended Readings

• Austin, Robert. (1996), Measuring and Managing Performance in Organizations (BOOK)

• James Bach: Heuristic Test Strategy Model
• Rex Black: Factors that influence test estimation (WEBSITE )
• Michael Bolton: Meaningful metrics
• David Goldberg: What every computer scientist should know about floating-point arithmetic
• Douglas Hoffman: The darker side of software metrics
• Cem Kaner and Walter P. Bond: Software engineering metrics: What do they measure and how do we know?
• Cem Kaner: Negotiating testing resources: A collaborative approach
• Cem Kaner: Recruiting software testers
• Michael Kelly: Using heuristic test oracles
• Michael Kelly: Estimating testing using spreadsheets
• Billy V. Koen: The engineering method and the heuristic: A personal history ("This was the beginning of a 37 year quest to find one thing that was not a heuristic.")
• Koen, Billy V. Definition of the Engineering Method, American Society for Engineering Education (ASEE). (A later version that is more thorough but maybe less approachable is Discussion of the Method, Oxford University Press, 2003) (BOOK)

• Jonathan Kohl: How do I Create Value with my Testing?
• Brian Marick: Experience with the cost of different coverage goals for testing
• Petzold, Charles. (1993), Code: The Hidden Language of Computer Hardware and Software. Microsoft Press (BOOK)

• Popper, Karl (2002, 3rd Ed.) , Conjectures and Refutations: The Growth of Scientific Knowledge (RoutledgeClassics). (BOOK)

• Erik Simmons: When will we be done testing? Software defect arrival modeling using the Weibull distribution
• Elaine J. Weyuker: On testing nontestable programs

Assignments

• Orientation Exercise for lecture on Impossibility of Complete Testing
• Mission of Testing
• Orientation Exercise for lecture on Oracles
• Oracle Heuristics 3-Phase Lab

Study Guide

• Foundations Study Guide