If you have ever rolled a feature out to 10% of users, you have already run an A/B test. You just did not measure it. For a few days, one group of users had the new checkout and the rest had the old one. That is a controlled experiment sitting right there in production, and most teams let the data evaporate. They watch the error rate, see nothing on fire, and slide to 100% without ever learning whether the feature helped anyone.
A/B testing with feature flags closes that gap. The same feature toggle that controls who sees the code can also record which side each user landed on and what they did next. You do not need a separate split testing tool, a tag manager snippet, or a second SDK. You need a hypothesis, a metric, and a flag that assigns users consistently. This post walks through how to do that without fooling yourself.
Every percentage rollout is an unmeasured experiment
A gradual rollout answers one question: does the new code break anything? That is worth answering, and we cover the mechanics in the gradual rollout post. But "it did not break" is a low bar. The more interesting question is whether the feature moved a number you care about, and a rollout can answer that too, for free, if you treat the off group as a control group.
The difference between a rollout and an experiment is not infrastructure. It is discipline. An experiment has three things a plain rollout does not:
- A hypothesis written down before you start. "The new onboarding flow will raise activation because it asks for less information up front." If you cannot finish the sentence "because...", you are not ready to test.
- One primary metric. The single number that decides the outcome. Signup completion rate, checkout conversion, day-7 retention. One. If you pick five, one of them will look good by chance and you will ship on noise.
- Guardrail metrics. Numbers that must not get worse: page load time, error rate, refund rate, support tickets. A variant that lifts conversion 3% while doubling latency is not a winner.
Write all three down before you flip anything on. Thirty seconds of writing prevents weeks of motivated reasoning later.
Assignment: consistent hashing and sticky cohorts
The heart of flag-based testing is how users get assigned to a variant. The naive approach, rolling a random number on every request, is broken in a subtle way: the same user flips between variants across page loads. They see the new pricing page, refresh, see the old one, and your data is garbage because nobody experienced either variant cleanly.
The fix is consistent hashing. Take a stable user identifier, combine it with the flag key, hash it, and map the result onto a 0 to 100 range:
bucket = hash("checkout.one-page" + user_id) % 100
if bucket < 50: variant = "B" else: variant = "A"Two properties fall out of this, and both matter:
- Stickiness. The same user always hashes to the same bucket, so their experience never flickers. Your cohorts stay clean for the whole test with no session storage and no database lookup.
- Independence between flags. Because the flag key is part of the hash, the users in variant B of
checkout.one-pageare not the same slice as variant B ofonboarding.v2. Without this, every experiment you run contaminates every other one.
This is the same mechanism a percentage rollout uses, which is exactly why a feature flags platform can do both jobs with one primitive. The rollout slider and the experiment split are the same hash with different intent. You can see the assignment flow end to end on our how it works page.
One practical warning: pick your identifier before the test starts. Anonymous visitor ID and account ID are both fine, but switching mid-test reshuffles everyone into new buckets and resets your experiment to zero.
How big, and how long
Sample size is where most flag-based tests quietly die. The rule of thumb: the smaller the effect you want to detect, the more users you need, and the relationship is brutal. Detecting a 20% relative lift might take a few thousand users per variant. Detecting a 2% lift can take hundreds of thousands. As a rough guide:
| Baseline conversion | Lift you want to detect | Rough users per variant |
|---|---|---|
| 5% | 20% relative (5% to 6%) | ~8,000 |
| 5% | 10% relative (5% to 5.5%) | ~31,000 |
| 20% | 10% relative (20% to 22%) | ~6,500 |
These are ballpark figures for a standard two-sided test; use a sample size calculator with your real numbers before you start. The point is the shape: small expected effects on low-traffic pages need more users than you probably have.
Then there is duration. Run for at least one full week, and preferably two, even if you hit your sample size early. Weekday users and weekend users behave differently, and a test that only saw Tuesday through Thursday is measuring a different population than your real one.
The peeking problem, in plain language
Here is the trap almost everyone falls into. You check the dashboard on day two, variant B is up 12%, the difference is "statistically significant," and you call it. The problem: significance calculations assume you look once, at the end. If you check every day and stop the moment the number crosses the line, you will find a "significant" result in tests where nothing is actually different, far more often than the stated 5%. Random noise wobbles above and below the significance line constantly during a test; peeking means you catch it on an upswing.
The fix is boring and effective: decide your sample size and duration up front, then do not decide anything until you get there. Look at the dashboard all you want, but only for guardrails. If errors spike or latency doubles, stop the test with the kill switch. That is safety, not peeking. Ending early because the primary metric looks good is peeking.
Reading results without fooling yourself
Novelty effects. New things get clicked because they are new. A redesigned button often wins its first week just by being different, then the lift fades as users habituate. If your effect shrinks steadily over the test window, extend the test. Retention metrics are more resistant to novelty than click metrics.
Segment fishing. The overall result is flat, so you slice by platform, country, browser, and plan until you find one segment where variant B "won." Slice enough ways and one segment will always look great by pure chance. Segments are for generating your next hypothesis, not for rescuing this one. If mobile users in one market seem to love the variant, run a new test targeted at that segment and see if it replicates.
Respect the flat result. "No detectable difference" is a finding, and often a valuable one: it means you can pick the cheaper, simpler variant with a clear conscience. Suppose the old flow converts at 4.5% and variant B comes in at 4.6% with wide error bars. That is not a mandate to ship B. It is permission to delete whichever version costs more to maintain, a point the flags vs branches post makes about dead code paths in general.
When not to A/B test
- Tiny traffic. If the table above says you need 30,000 users per variant and you get 500 signups a month, the test will take five years. Ship it as a gradual rollout with guardrail monitoring, talk to users, and move on.
- Bug fixes. The broken behavior is not a variant worth preserving. Fix it, roll it out, done.
- Compliance and security work. You are not going to keep the non-compliant version if it converts better. There is no decision for the data to inform.
- One-way doors. Data migrations and pricing changes that are painful to reverse deserve prototypes and user research, not a coin flip on live traffic.
Ship the winner with the slider you already have
This is the part that makes flag-based testing better than standalone A/B testing software. In a separate testing tool, "variant B won" produces a ticket: rebuild the winning variant in the actual product, redeploy, hope the rebuilt version matches what was tested. With flags, variant B already is production code. Shipping the winner means dragging the rollout slider from 50% to 100%, watching guardrails on the way, and then deleting the losing code path and the flag itself. The experiment, the rollout, and the cleanup are one continuous motion on one checkout.one-page flag.
This is the model Toggled is built around: experimentation as a property of every flag, not a separate analytics contract. Toggled is in early access, with flat per-team pricing and unlimited seats, so a bigger team does not mean a bigger bill (details on the pricing page). If you want to see the loop before signing up for anything, the homepage demo includes an illustrative A/B strip showing how a split and its results sit right next to the rollout controls.