Forum Discussion
Do you use control groups in your Canvas set up?
Short answer - yes. However, I do agree with alextoh1 that if you're dealing with really marginal volumes, you won't get much insight from a control group. It comes down to your goals and strategy as well - i.e. whether you're focusing on test-learn-iterate and if you need to measure/report on the lift/impact of your CRM efforts, for example, for decision-making, prioritization, reporting to stakeholders, fine-tuning your stratgey etc.
Control groups are good to understand:
- the effectiveness of a specific campaign (campaign / canvas level) at driving your KPI --> campaign/canvas level control group (I typically go for 10% but it would depend on how large your base is, as you'd want to ensure you reach statistical significance)
- measure an experiment's variant performance --> A/B or multi-variate (depends on test, target size, MDE, etc - recommend to use a calculator like abtestguide or optimizely but there are a number; also great to work with your analytics team on this!).
- overall performance of your CRM strategy --> Universal Control Group (UCG) set up at the account level (really depends on your userbase size - Braze has some good guidelines here)
The exceptions is typically Transactional/Service communications. But there could be other reasons why you choose to exclude the control/UCG - dependant on your own strategy/needs.For example, I once had a client testing different onboarding flows through Braze (using a sophisticated IAM flow setup) due to lack of dev resources at the time to test more natively in the product or a better Product testing tool - they chose to exclude the UCG since they deemed necessary to ensure 100% of the base got exposure to some variant of an onboarding.
On your question about the puzzling incremental lifts - could be, but you need to understand whether your audience sizes and test duration are enough to make your results stat sig; whether you're confident the variable you're testing can directly impact your KPI or if there are other campaigns and external factors that could be playing a role; ... Again, it's always good to work closely with an analyst in planning and measuring your control group strategy and specific experiment results!
Lastly, I do think it's worth using incremental uplift when talking about performance - if you have a clean, significant test, with a high confidence that the difference observed is affected by the variable changed, you will have a very clear case for applying it and optimizing your strategy. You will be making data-driven decisions, and optimizing your strategy and resource allocation based on concrete evidence. In my experience, stakeholders can easily understand this and the concept of incremental lift (it might require some education at first, but it will ultimately open up more interest and awareness around the importance of your strategy and, consequently, make it easier with resources).
Related Content
- 3 months ago
- 3 months ago
- 2 years ago