Lesson 9: Quality Assurance
To test whether your code works as intended, use override rules to assign specific users to your new feature. You can also run experiments on employees only, and run A/A tests to test your setup end-to-end before launching your main experiment.
Overrides
You can assign a specific user to a particular treatment by overriding the randomization. This means that you can add yourself or other members of the experimenting team to a specific variant at any time to try it out. You can verify that your implementations appear to be working as they should before releasing the experiment to actual users.
Overriding users into specific treatments doesn't affect the results, as the exposure data doesn't include the overrides.
In Confidence, you create override rules to assign specific users to a particular treatment.
Employee only
Depending on the nature of your product, a powerful next step in the QA process is to run your experiment on employees only. Make sure to include an attribute in the evaluation context that identifies the incoming request as belonging to an employee, and then use that in your inclusion criteria. This way, you can test your change and its different values on users that are a bit more forgiving. It can give you the chance to detect errors that you might not notice during the early stages of QA. The drawback is of course that the sample size is typically so small that it's difficult to find any meaningful effects, but you might hear from your colleagues if something isn't working as it should.
For employee experiments to be possible you must include employee status in the evaluation context of your feature flag.
You can also give your new feature to employees only by directly creating a rule on your flag that has employee status as an inclusion criteria.
A/A tests
Sometimes you may want to run an A/A test to test your overall setup before launching the actual experiment. An A/A test is just like an A/B test, except that the experiences given to the control and treatment groups are the same. Either the two variants you use are the same, or you resolve the flag in your code but don't use the received variant values. A/A tests are particularly helpful if you want to test your whole setup end-to-end on real users and get real exposure data.
A/A tests are useful when you have just integrated your service with Confidence. The A/B test quickstart describes such a test.