anomalyDetectorContextual

This example uses a dataset that is far too large to display fully here. Instead, we've uploaded the data into THIS DOWNLOADABLE PROJECT. The project contains no script content, so you can copy-paste in the example ATL and try it yourself.

The full dataset contains sales data for 2018 through 2021. For reference, the first six rows are:

Suppose you want to look for scale anomalies by comparing the sales data for equivalent months. For example, you might want to know if the sales total for June 2021 is unusually high or low compared to June sales in other years. Also, by unusually high or low, you mean values that are 40% higher or lower than usual, allowing for normal variation (as measured by standard deviation).

The ATL is: [[anomalyDetectorContextual(Sales, Date, 'monthly', 40)]] and the result is.

The results list contains five ATL objects. There are three scale anomalies and two directional anomalies. For this example, we want to focus on the scale anomalies. You can filter by anomalyType to leave the scale anomalies only:

[[filter(anomalyDetectorContextual(Sales, Date, 'monthly', 40), x -> x.anomalyType == 'scale')]]

The filtered result is:

There are three scale anomalies: one high and two low. The anomalous values are the aggregated totals for Mar 2019, Jun 2020, and Dec 2020. Note that the date value in each object is the date for the first value during these periods.

Note also that the detected values do not represent the highest or lowest aggregated totals in the dataset. They are anomalies because they are unusually high or low for a specific month. This is easier to see by looking at this table of aggregated totals:

The aggregated total for Mar 2019 (70,000) is not the highest overall but it's unusually high compared to the equivalent totals for 2018, 2020, and 2021. Similarly, the values for Jun 2020 and Dec 2020 are anomalous because they are unusually low for their group.

Here's how you might extract values from the result to produce narrative text:

[[

results = anomalyDetectorContextual(Sales, Date, 'monthly', 40)

scaleAnomalies = filter(results, x -> x.anomalyType == 'scale')

highOnly = filter(scaleAnomalies, x -> x.anomalyClass == 'high')

lowOnly = filter(scaleAnomalies, x -> x.anomalyClass == 'low')

highValues = map(highOnly, x -> "[[formatDateTime(x.date,'MMMM yyyy')]] ([[abbreviateNumber(currencyFormat(x.value))]])")

lowValues = map(lowOnly, x -> "[[formatDateTime(x.date,'MMMM yyyy')]] ([[abbreviateNumber(currencyFormat(x.value))]])")

yearRange = joinStrings('the years ', unique(Year)[ :1], ' through ', unique(Year)[-1: ])

"Comparing equivalent months for [[yearRange]], total sales were unusually high in [[highValues]] and unusually low in [[lowValues]]."

]]

The output text is:

In Example 1, the function initially found a mixture of scale and directional anomalies. For simplicity, we filtered the result and focused on the scale anomalies only. In this example, we'll take a closer look at the directional anomalies.

For this example, assume a "Describe the Table" project with the same data used for Example 1.

[[filter(anomalyDetectorContextual(Sales, Date, 'monthly', 40), x -> x.anomalyType == 'directional')]] returns this result:

There are two directional anomalies: one in Jun 2020 and the other in Dec 2020. With directional anomalies, the anomalyClass value is either up or down. For both anomalies here, the value is down.

It's easier to understand these anomalies by looking at this table of aggregated totals:

The Jun 2020 total (10,000) is anomalous because it's down from the preceding month's total (28,000), which makes it unique for its group. Note that the May-to-Jun direction is up for all other years in the dataset. The Dec 2020 total (13,000) is anomalous for the same reason: in all other years, the Nov-to-Dec direction is up rather than down.

Here's how you might extract values from the result to produce narrative text:

[[

results = anomalyDetectorContextual(Sales, Date, 'monthly', 40)

dirOnly = filter(results, x -> x.anomalyType == 'directional')

dirOnlyCount = len(dirOnly)

dirDetails = map(dirOnly, x -> "[[formatDateTime(x.date,'MMMM yyyy')]] (sales [[x.anomalyClass]])") 

yearRange = joinStrings(unique(Year)[ :1], ' through ', unique(Year)[-1: ]) 

"The analysis compared equivalent months for [[yearRange]] and found [[numToWords(dirOnlyCount)]] directional [[inflectNoun('anomaly',dirOnlyCount)]]. [[if(len(dirOnly) > 0){The [[inflectNoun('anomaly',dirOnlyCount)]] occurred in [[dirDetails]]}]]."

]]

The output text is:

Assume a "Describe the Table" project with this data:

The table contains sales data for Feb 2021 only. Typically, you would have sales data covering a longer period, but we've kept the dataset small to simplify the example. Use THIS DOWNLOADABLE PROJECT if you want to try the example yourself.

Suppose that you want to know if sales on a particular Monday performed significantly better or worse than on other Mondays — and also run the same analysis for all other weekdays. Also, suppose that significantly better or worse means values that are 30% higher or lower than usual, allowing for normal variation (as measured by standard deviation).

The ATL is: [[anomalyDetectorContextual(Sales, Date, 'daily', 30)]] and the result is:

There are two scale anomalies, one high and one low. Note that the detected values are not the highest and lowest in the Sales column. They are anomalies because they are unusually high/low for a specific day of the week.

This is easier to see by looking at this table, which shows the breakdown for each week by day:

The high scale anomaly occurred on Feb 8, 2021, which is a Monday. The anomalous value (12,000) is not the highest aggregated value in the table, but it's unusually high when compared to the aggregated totals for the other Mondays (Feb 1, Feb 15, and Feb 22).

Similarly, the low scale anomaly occurred on Feb 25, 2021, which is a Thursday. The anomalous value (10,000) is not the lowest aggregated value in the table, but it's unusually low when compared to the other aggregated totals in its comparison group.

Here's how you might extract values from the result to produce narrative text:

[[

global.results = anomalyDetectorContextual(Sales, Date, 'daily', 30)

global.scaleOnly = filter(results, x -> x.anomalyType == 'scale')

global.highScale = filter(scaleOnly, x -> x.anomalyClass == 'high')

global.lowScale = filter(scaleOnly, x -> x.anomalyClass == 'low');''

]]

[[map(highScale, x -> "sales on [[x.date]] ([[abbreviateNumber(currencyFormat(x.value))]]) were unusually high for a [[formatDateTime(x.date,'EEEE')]] (normal average = [[abbreviateNumber(currencyFormat(x.normalMean))]])")]].

[[map(lowScale, x -> "sales on [[x.date]] ([[abbreviateNumber(currencyFormat(x.value))]]) were unusually low for a [[formatDateTime(x.date,'EEEE')]] (normal average = [[abbreviateNumber(currencyFormat(x.normalMean))]])")]]. 

The output text is: