Personalization Statistics

Comparing statistics between personalizations and variants is a good way to evaluate personalization and variant performance, and gain valuable actionable insights.
For this reason we keep working on statistics that will work for you. This short article will help you on your way to interpreting the available statistics.

Concepts & terminology

Attribution events

Events that will be attributed to a personalization variant as conversions, good examples for example are Purchase and EmailOptIn events.
Any attribution event is counted as a conversion if the customer has touch points with the personalization in question (via attributable events) within the attribution window.

A single attribution event can be attributed to multiple personalizations.
Unique order id’s will be counted towards conversions in the case of Purchase attributions.

Attributable events

Personalization statistic events to which an attribution event can be linked (currently PersonalizationView, PersonalizationClick and PersonalizationClosed).

Attribution models

Last Interaction
Using this model, an attribution event can only be attributed to one variant, per personalization. The last touchpoint will receive the conversion.
This means only the most recent variant is attributed if a customer sees multiple variants within the same attribution window.
A customer can see multiple variants, if for example using multiple browsers, computers, or after clearing cookies.

Caveats
Uplift values on personalization level
It can occur that uplift values shown in the personalization row are unexpectedly higher than the variant rows summed.
This occurs because they are calculated using unique visitors.
The personalization's visitors are unique for the entire personalization, visitors for variants are unique per variant, however.
If a customer sees multiple variants this will cause the sum of all the visitors from the variant rows to differ from the total number of visitors displayed in the personalization row.
This results in the uplift value on personalization level to seem higher than expected because the number of unique visitors in the personalization row will be lower than the sum of all visitors displayed per variant.

Linear
When displaying statistics using the linear model, all variants a customer interacted with will be attributed. As long as the touchpoint is within the chosen attribution window they will count as conversions.
The personalization level statistics will be attributed on personalization basis, not on variant basis. This means the numbers there can be different from the statistics of all variants combined.

Caveats
When using the linear model, visitors are not necessarily unique per personalisation. A single customer can be attributed to two variants within the same personalisation. Unique visitors are de-duplicated on personalisation level. This causes the conversion rate to be higher on personalisation level, and the uplift will not relate to the variant rows.
We always display the Last Interaction model in the personalisation row to prevent this.

Attribution windows

Currently we offer two different attribution windows.
Attribution events will either be counted towards conversions if within 1 day after a view or within 7 days for clicks for the default window.
Using the alternate window this is extended to 7 days for views and 28 days for clicks.

One thing worthy of note, is that conversions will continue to be counted even after the personalization has been paused.
Conversion numbers can increase until the attribution window has completely passed since the last received statistic events.

Which statistics are available, and how they are calculated?

Please keep in mind that the control group variant, if present, will always be excluded from the personalization level basic statistics.

Basic statistics:

• Visitors:
  Number of unique views calculated with PersonalizationView events

• Views:
  Total number of PersonalizationView events

• Clicks:
  Total number of PersonalizationClick events

• Conversions:
  The number of conversions, calculated using the selected attribution event(s) and window

Derived statistics:

• Click-through rate:
  The rate of clicks to views.
  Calculated using: [clicks / views * 100]

• Conversion rate:
  Calculated using: [conversions / visitors * 100]

• Conversion value:
  Calculated using: [Total conversion revenue]

• Average order value:
  Calculated using: [Conversion value / conversions]

• Order value per visitor:
  Calculated using: [Conversion value / visitors]

Derived variant comparison statistics:

These statistics refer to relative differences between variants within the same personalization.
They can be used to more easily judge variant performance.

• Added conversion value:
  Calculated using: [Conversion value variant - variant Visitors * Order value per visitor of the control group]

• Conversion rate uplift:
  This is the uplift compared to the control group.
  Calculated using: [(Conversion rate variant - Conversion rate control group) / Conversion rate control group * 100]

• Average order value uplift:
  This is the uplift compared to the control group.
  Calculated using: [(Avg. order value variant - Avg. order value control group) / Avg. order value control group * 100]

• Order value per visitor uplift:
  This is the uplift compared to the control group.
  Calculated using: [(Order value per visitor variant - Order value per visitor control group) / Order value per visitor control group * 100]

• Total value uplift:
  This is the uplift relative to the total sum of added conversion values of the personalization variants, excluding the control group.
  Calculated using: [Added Conversion value variant / Conversion value complete personalization (excluding control group) * 100]

Caveats

Influence of settings on statistics

There are some variant settings that can impact the comparability of some specific statistics between variant statistics.
Using these in an inconsistent way will result in unfair comparisons.
One example is when the variant trigger frequency setting is used. If one variant is shown once every x days, while another is always shown it is possible the variant that is always shown will collect more conversions.
This is due to the attribution window being 'extended' after every interaction.
When this happens it might skew any comparisons between variants because the conversion rate is calculated using unique visitors.

Minimum requirements

Keep in mind that personalizations need at least 100 views before statistics will be shown.

Purchase attributed personalizations

The orderid in purchase events will be used to identify unique conversions. If only empty orderid fields are sent with purchase events it’ll cause all purchase attributed personalizations to show a maximum of 1 conversion. If irregularly sent, it’ll cause the conversions to be irregularly counted as well.

The future

If deemed as a useful feature, we might make a setting available in the future with which the attribution window can be chosen freely (within reasonable limits).

If you could use a specific derived statistic that is missing, don't hesitate to contact us!
We can always use your suggestions to see if we can expand our statistic overviews with more useful information when it is useful for everyone.