Analysis Quickstart

​

Before You Begin

To run an analysis, you need:

• an entity
• a fact table
• a metric

Use the metrics quickstart to create an entity, fact table and a metric.

​

Step 1: Create the Assignment Table

An assignment table is a table that has data about assignments of entities to variants in experiments. For your analysis to be able to read the data, you need to create an assignment table in Confidence. Follow the steps on the assignment table page to create your table.

​

Assignment Table from Optimizely Decision Events

You can analyze past or current experiments in Optimizely with Confidence. To do so, you need to export the decision events from Optimizely to a table in your data warehouse. A decision event is an event that Optimizely records when a visitor is exposed to an experiment. Decision events in Optimizely correspond to assignments in Confidence. The information Confidence requires is available in the columns:

• experiment_id: column with identifiers of the experiments
• variation_id: column with identifiers of the variants
• visitor_id: column with identifiers of the entities in the experiments, like users and visitors
• timestamp: column with timestamps of the events

To set up the assignment table in Confidence, follow these steps:

1. Export the decision events from Optimizely to a table in your data warehouse. If you want to analyze a running experiment, you need to schedule the export to happen at a regular cadence.
2. If you don’t have one already, create an entity in Confidence that identifies the entity that’s recorded in the visitor_id column of the decision events table.
3. Create an assignment table in Confidence that points to the decision events table.
   • Set the exposure key column to experiment_id.
   • Set the variant key column to variation_id.
   • Set the entity to the entity you created in step 2.
   • Set the entity column to visitor_id.
   • Set the timestamp column to timestamp.

The columns experiment_id and variation_id must be strings to be selectable as exposure key and variant. The type of the visitor_id must match the primary key type of the entity you created in step 2, such as a string. The timestamp column must be a timestamp.

​

Step 2: Create an Analysis Workflow

Open Confidence and select Analyses on the left sidebar. The overview page shows all draft, live, and ended analyses that you have permission to view. Click + Create in the upper right corner to create a new analysis.

​

Step 3: Name, Owner and Assignment Table

Specify which entity the experiment used for randomizing the treatment assignment, for example ´User´. Confidence uses the entity to map the metrics to the units in the analysis. You first need to give your analysis a name and assign an owner. Use a descriptive name that others understand. For this exercise, use:

• Name: analysis-<your-name-and-date>
• Assignment table: If you have more than one assignment table in Confidence, select which assignment table to use for this analysis.
• Entity: Select the entity that represents the unit you experiment on.
• Owner: Select yourself

Click Create. You’re now on the analysis design page.

​

Step 4: Dates and Exposure Key

Select the date range for the experiment to avoid that the queries scan unnecessary data. The analysis start date is the first day of the experiment that you want to analyze. If the experiment started on 2023-01-01, input 2023-01-01 as the start date to begin analyzing it on the day it started. Enter the identifier of the experiment in the assignment data as the exposure key. For example, if your experiment identifier is experiment-123, enter experiment-123 as the exposure key. If you’re analyzing an experiment based on decision events exported from Optimizely, your exposure key is the identifier for your experiment that’s available in the experiment_id column in your decision events table.

​

Step 5: Treatments

To configure the treatment groups for your analysis, you need to enter the variant keys for the treatment groups. Confidence pre-populates the treatment variant list with the unique values found in the variant key column of the assignment table for the selected exposure key. The variant key is the identifier for each group in the experiment that, together with the exposure key, uniquely identifies the relevant group in the assignment table. For example, if the control group is default-style, enter default-style as the variant key. To set up your treatments:

​

Step 6: Metrics

To measure the outcome of the experiment, you need to add metrics to the analysis. In Confidence, metrics are either Success metrics or Guardrail metrics. Add your metric as a success metric if you hope to see an improvement in the metric. For example, with your change you hope to see an increase in the number of purchases per user. Add your metric as a guardrail metric if you don’t expect to see a change, but you want to make sure the change doesn’t have a negative impact. For example, with your change you don’t want to see an increase in the number of returned items per user. To add your metrics to the analysis: Read more about minimum detectable effects and non-inferiority margins. Your analysis can have required metrics added from the surface that the analysis belongs to. Read more about required metrics.

​

Step 7: Calculate

Review your setup and click Calculate to run the analysis. You are now on the Result page. You can add exploratory analyses on the Result page. Click Explore on the Metrics result section.

​

Related Resources