Regression

When there is a new version of your system or when changes have been made to it, you must run regression tests to check that all the previous text and functionality still work (i.e. they have not regressed). Automated tests are excellent for this.

Automation. It is common to develop the text incrementally in the shape of initial sentences, then subsequently introduce variation or complexity – through lex rule groups, aggregation, or something else. So if you automate tests for regression right from the start, you will probably find that you have to keep adapting them, and you can spend a lot of time checking test failures to see whether they are the result of a deliberate change to that bit of text or whether they are actually an inadvertently introduced error (which is what regression tests are supposed to find).

You may choose to only develop the automated tests once all variation has been introduced. But this means you will have a lot of tests to automate later in the project, and you will have to test a lot manually in the earlier phases of the project.

Deciding when to test. One way (already mentioned) to avoid this is to develop the text incrementally by fully developing one piece of text at a time (e.g. one paragraph or section fully developed with variation, then another one added, then another, and so on). Obviously this requires agreement with the development team and careful planning of user stories. Depending on what you’re developing, it may not be as easy as it sounds.

Multiple variations need to pass. Remember that variation means that running the same test (same data, same pre-conditions) may produce different output each time (i.e. the varying text). So the automated test needs to pass if any of the correct variations are produced, but not if another variation occurs.

Regular expressions are a useful way of dealing with these. Regular expressions (regexp) are a way of finding text strings that match an expression that defines the expected matching text, including allowed variations. For a simple example:

Get Regexp Matches ${outputSentence}Temperatures are expected to ((stay|remain|hold) above zero)|(continue above freezing) overnight

In the example above, variations are bounded by the | character and grouped by parentheses.

So this checks that the output sentence will match if it says Temperatures are expected to stay above zero overnight OR Temperatures are expected to continue above freezing overnight. Obviously this is not the only way to approach this — you could use loops against a set of variations, or a set of if-else statements.