Comparing statistics between personalizations and variants is a good way to evaluate personalization and variant performance, and gain valuable actionable insights. For this reason we keep working on statistics that will work for you. This short article will help you on your way to interpreting the available statistics.
Concepts & terminology
Events that will be attributed to a personalization variant as conversions, good examples for example are Purchase and EmailOptIn events. Any attribution event is counted as a conversion if the customer has touch points with the personalization in question (via attributable events) within the attribution window.
A single attribution event can be attributed to multiple personalizations. Unique order id’s will be counted towards conversions in the case of Purchase attributions.
Personalization statistic events to which an attribution event can be linked (currently PersonalizationView, PersonalizationClick and PersonalizationClosed).
Last Interaction Using this model, an attribution event can only be attributed to one variant, per personalization. The last touchpoint will receive the conversion. This means only the most recent variant is attributed if a customer sees multiple variants within the same attribution window. A customer can see multiple variants, if for example using multiple browsers, computers, or after clearing cookies.
Caveats Uplift values on personalization level It can occur that uplift values shown in the personalization row are unexpectedly higher than the variant rows summed. This occurs because they are calculated using unique visitors. The personalization's visitors are unique for the entire personalization, visitors for variants are unique per variant, however. If a customer sees multiple variants this will cause the sum of all the visitors from the variant rows to differ from the total number of visitors displayed in the personalization row. This results in the uplift value on personalization level to seem higher than expected because the number of unique visitors in the personalization row will be lower than the sum of all visitors displayed per variant.
Linear When displaying statistics using the linear model, all variants a customer interacted with will be attributed. As long as the touchpoint is within the chosen attribution window they will count as conversions. The personalization level statistics will be attributed on personalization basis, not on variant basis. This means the numbers there can be different from the statistics of all variants combined.
Currently we offer two different attribution windows. Attribution events will either be counted towards conversions if within 1 day after a view or within 7 days for clicks for the default window. Using the alternate window this is extended to 7 days for views and 28 days for clicks.
One thing worthy of note, is that conversions will continue to be counted even after the personalization has been paused. Conversion numbers can increase until the attribution window has completely passed since the last received statistic events.
Which statistics are available, and how they are calculated?
Please keep in mind that the control group variant, if present, will always be excluded from the personalization level basic statistics.
Visitors: Number of unique views calculated with PersonalizationView events
Views: Total number of PersonalizationView events
Clicks: Total number of PersonalizationClick events
Conversions: The number of conversions, calculated using the selected attribution event(s) and window
Click-through rate: The rate of clicks to views. Calculated using: [clicks / views * 100]
Average order value: Calculated using: [Conversion value / conversions]
Order value per visitor: Calculated using: [Conversion value / visitors]
Derived variant comparison statistics:
These statistics refer to relative differences between variants within the same personalization. They can be used to more easily judge variant performance.
Added conversion value: Calculated using: [Conversion value variant - variant Visitors * Order value per visitor of the control group]
Conversion rate uplift: This is the uplift compared to the control group. Calculated using: [(Conversion rate variant - Conversion rate control group) / Conversion rate control group * 100]
Average order value uplift: This is the uplift compared to the control group. Calculated using: [(Avg. order value variant - Avg. order value control group) / Avg. order value control group * 100]
Order value per visitor uplift: This is the uplift compared to the control group. Calculated using: [(Order value per visitor variant - Order value per visitor control group) / Order value per visitor control group * 100]
Total value uplift: This is the uplift relative to the total sum of added conversion values of the personalization variants, excluding the control group. Calculated using: [Added Conversion value variant / Conversion value complete personalization (excluding control group) * 100]
Influence of settings on statistics
There are some variant settings that can impact the comparability of some specific statistics between variant statistics. Using these in an inconsistent way will result in unfair comparisons. One example is when the variant trigger frequency setting is used. If one variant is shown once every x days, while another is always shown it is possible the variant that is always shown will collect more conversions. This is due to the attribution window being 'extended' after every interaction. When this happens it might skew any comparisons between variants because the conversion rate is calculated using unique visitors.
Keep in mind that personalizations need at least 100 views before statistics will be shown.
Purchase attributed personalizations
The orderid in purchase events will be used to identify unique conversions. If only empty orderid fields are sent with purchase events it’ll cause all purchase attributed personalizations to show a maximum of 1 conversion. If irregularly sent, it’ll cause the conversions to be irregularly counted as well.
If deemed as a useful feature, we might make a setting available in the future with which the attribution window can be chosen freely (within reasonable limits).
If you could use a specific derived statistic that is missing, don't hesitate to contact us! We can always use your suggestions to see if we can expand our statistic overviews with more useful information when it is useful for everyone.