Glossary of testing terminology

Automation

Employing automated scripts to test outputs. A report is then generated with a breakdown of pre-set tests that pass and fail.

Black-box testing

Testing a system without knowledge of how the system is built. The user is concerned only with the expected and actual behavior of the system.

Exploratory testing

Using and exploring the system without a specific plan or set of expected outputs; looking to see how it works and what goes wrong. A form of exploratory testing is destructive testing, in which the user tests challenging scenarios for the system with the intention of seeing how it can be pushed before breaking.

Manual testing

Testing by a human user, without the aid of automation. Manual testing is sometimes confused with exploratory testing. Exploratory testing is done manually, but exploratory testing and manual testing are not the same thing.

Regression

Testing a new version with the aim of ensuring that all existing functionality from the previous version is intact and has not been compromised by recent changes.

Test case

A description (or specification) of a given set of input data + preconditions + actions, along with the consequent output and actions that those should produce. Test cases need to test behavior on unusual edge cases as well as normal situations, because this is where failure is most likely.

User acceptance testing

Testing carried out by, or on behalf of, the owner and end users of the system in order to determine whether the system is acceptable. Does it perform its agreed functions? Does it produce the desired outputs?

Validation testing

Testing that a system meets the needs of the users. This means testing that it works the way users want it to work. In an NLG system, validation testing checks that the language the system uses is satisfactory.

Verification testing

Testing that a system performs as designed.

White-box testing

Testing a system with knowledge of how the system is built.