Metrics and Dimensions
To produce results, you need to add metrics to your analysis. You can select any metric that uses the same entity that you configured in the metric configuration section for your experiment. For each metric, you can add dimensions (if they exist). Dimensions come from dimension tables that use the same entity as the metric. Use dimensions to find out if the experiment had a different impact on different subgroups.Static Dimensions and Dimensions that Change over Time
Confidence supports both static dimensions and dimensions that change over time. Static dimensions don’t change rapidly over time and aren’t affected by the experiment treatment. For example, user age, registration country, preferred language, etc. If the experiment treatment can affect a dimension value, it’s a time-changing dimension. For example, user’s subscription status, consumption patterns, feature usages, etc. To ensure that segmentation is not misleading for time-changing dimensions, Confidence always uses the dimension value at the time right before the user got exposed to the treatment. This way, the treatment doesn’t influence the dimension value itself.If users don’t have dimension values in the table they get NULL as
dimension values. Make the result interpretations easier by mapping NULL values
to a meaningful value in the dimension table configuration.
Examples of Dimensions That Change over Time
New User vs Existing User
The dimension table has a columnis_existing_user that has a true value for
all existing users at a given time. Since new users that come in during the
experiment don’t have a value on this dimension table before exposure by
construction, they get the value NULL. You can map the NULL value to “New user”
in the dimension-table configuration to make the interpretation of the
exploration easier.
You also need to decide on when to consider a user “New” relative to their first
exposure. For example, you can use a 1 day window if a user needs to have signed
up within 24 hours before exposure to be in the “New” category.
User that Took a Certain Action the Week Before the Experiment
Examples of this action could be users that watched a video, added something to checkout, used a certain feature in your app, etc. The dimension table has a columntook_action_x_last_7_days that captures a
rolling seven day window and only uses the last value. If you produce this data
hourly, you can use a one hour window in the window setting for the dimension in
the dimension-table configuration.
If your dimension value requires aggregations, perform those directly in the dimension table query or before. Confidence only uses the latest non-NULL value in the time window.
took_action_x with hourly values in the
dimension table and you configure Confidence to use a 7 days window to
fetch the dimension value, Confidence uses the last non-NULL value
in that 7 day window which doesn’t correspond to the goal of this example.
Stale Analyses
Sometimes, your analysis might end up in aStale state. If you hover the
status, you see a message describing why the analysis is stale. Two reason are:
- New data exists, and results might be different if you run the calculations again. To calculate new results based on all available data, click Retrigger above the results.
- The conclusion is older than the results. Retriggering result calculations because of the first reason is a common explanation for this. It means that the results might have changed and the conclusion is possibly invalid. Update the conclusion to get rid of this warning.