In this article we will look at the standard traffic allocation modes you can select when configuring your experiences and answer some FAQs.
Of course, you may choose to select a custom split for your experience. There's more about this in Traffic allocation.
INFO: It is worth noting that once the experience has been published, even if you subsequently pause it, you cannot change from a custom split to a standard mode traffic allocation mode.
DANGER: During the lifetime of an experiment, especially if an experiment shows a positive impact against the defined goals, there will be a temptation to change the traffic allocation. This will result in flipping. Increasing the traffic allocation—flipping users from the control to the variation, e.g. Pilot 20/80 to Split 50/50 —is a tolerable practice, but one we would not recommend just the same.
DANGER: Reducing the traffic allocation—flipping users from the variation to the control, e.g. Supervised 5/95 to Split 50/50— is not an acceptable practice, principally because it leads to misleading experiment results. For this reason, and others, we prevent changes to traffic splits that flip users from a variation into the control once the experience has been published. To prevent flipping in both directions, we recommend duplicating the experience and then setting the desired allocation, before pushing live.
Pilot mode is best used for testing high risk experiences and is often used to detect issues or bugs. The variation is only shown to 20% of users.
See this Qubit.Blog post for a discussion about why Pilot mode testing is important.
Split mode is the default traffic allocation setting because it is the most effective mode for A/B testing. The variation and control will receive an equal amount of users over time. Split mode reduces the amount of time between launching an experience and getting conclusive test results.
Supervised mode is typically used after Split mode, when your test is successful, and you wish to roll it out to a larger percentage of users.
Keeping a 5% control will ensure any long-term behavioral changes remain visible.
All traffic mode is designed for delivering content that has to be shown to all users, for example legal messages. Performance tracking is not possible in this mode.
Pilot tests run at much lower power (they don’t have much chance of detecting uplift), so they complete much faster. They should generally only be used to test that a change does not produce a massive negative effect.
A normal test requires 6,000 converters in each variant. If a customer has only a 5% chance of being put in the control, it will take a much longer time to get to 6,000 for that variant. 50/50 is the fastest possible A/B test.
Users will be reallocated according to the new split. It should work the same way as a regular change in traffic allocation.
The way it works is that every user is assigned a random number between 0 and 1, if the split is say 33, 33, 33, then visitors with numbers between 0 and 0.33 are in the control, between 0.33 and 0.67 in variant 1, and visitors over 0.67 in the second variant. If you change to 50/50 then visitors below 0.5 are in the control and visitors over 0.5 are in the variant.
So in this case, every one in the control at the start remains in the control, half the visitors in old variant 1 are placed into the new control, the other half in new variant 1, and all visitors in the old variant 2 will be placed in the new variant 1.
This can certainly happen if you change the traffic allocation for an experience whilst it's still running. This is referred to as flipping. If an experience starts in Pilot mode then 80% of users will be in the control. If you now switch to the 50/50 Split mode, 30% of the users will move from control to variant. If you later switch to the All traffic mode, then all the previous control visitors will naturally move to the variant.
The same may happen when an experience is run in All traffic mode for a period of time before being changed to Split mode (not a recommended change). In this situation, 50% of users will of course move from the variant to the control.
Users that acquire multiple Qubit Ids can also see both the control and variant. This typically happens on gaming sites, where some people use different devices, many browsers, or go incognito.
To reduce the number of users who enter both variations, it is possible to include a code snippet to check for
user_id in the experience triggers.
Finally, if a user sees an experience variation and later turns off cookies, they will then no longer see the variation.
DANGER: We do not recommend switching an experience from the All traffic mode to a split experience, as it makes the time to completion much longer. We recommend duplicating the experience and restarting it as a new one. That would also remove the issue of visitors flipping.
As mentioned, for many reasons, but principally in the interests of statistical robustness, we do not allow changes to splits that will result in users flipping from the variation into the control. Instead, we recommend duplicating the experience and re-running it with the desired split.