4 — Testing NLG Studio projects

Not so much! The general principles of testing are the same — you are trying to test that the behavior of the system corresponds to the design or the requirements (i.e. it behaves as intended under all the different scenarios of input data and user actions). However, you are also testing language, and there are different ways to say the same thing! In other words, varying sentences might be used in the same place and each might be correct. So the same test might have multiple acceptable results.

Linguistic variation poses different challenges. Variation in texts that is caused by variation in data can be treated in the same way as any other testing — each is a different test case. For example, Profits rose dramatically in the first quarter versus Profits rose slightly in the first quarter are sentences produced from data for two different years or two different companies. But variation in texts that reports on the same result, such as Profits rose dramatically in the first quarter or Profits dramatically increased in Q1 or Profits climbed massively in the first three months are all generated for the same set of input data (same company, same year).

Tests (especially automated tests) must allow for the fact that there are multiple correct answers. Variation can be tested in component tests, but it is important to test end-to-end to check that:

Language is subjective. What might be technically correct might not be acceptable to the user; it might not read well or it might not be quite the right terminology. (This is a question of validation rather than verification.) So keep in mind that the testing of NLG systems must focus on the quality of the produced texts, in addition to correctness (i.e. correct behavior). Does the terminology sound right to the user?

Here are a couple examples:

In the above sentence, feedback from a weather forecaster told us that the conjunction used should be and not but.

Actually, this should say Rain will turn to snow. The problem here was caused by values appearing in the narrative the other way around. When originally tested with data, the result was Snow will ease to rain overnight, which was fine!

Does the system use domain sub-language? For a given (non-public facing) system, language is likely to operate in a specialized domain, therefore the language itself is specialized. Here are some examples:

This means that when testing the system, particularly from a quality perspective, you will have to pick up some of the lingo! Advice from SMEs and/or the product owner is invaluable, as is lots of reading.

Moreover, we have found that this makes it very difficult to perform any “automated” grammar checking and style checking on the text, for example with a system such as Language Tool. Generally, such systems will highlight issues that actually are valid in your text (abbreviations, specialized terms, etc.) and in particular they will almost always be unable to spot language that is correct English but incorrect for the particular system domain.

When strategizing how to test an NLG system, there are special considerations.

Testing is usually done by running a system on test cases. Test cases need to test behavior on unusual edge cases as well as normal situations, because this is where failure is most likely.

In order to avoid repetitive narratives, we often add random variation to them, typically using Studio’s chooseAtRandom function. Testing that good narratives are produced in all variants can be challenging, so when adding variation, don't overdo it — especially if the client doesn't need much.

In testing NLG systems, the following are some things to consider: