The short answer: a feature flag is a switch that decides who sees a code path. An A/B test is a measurement that decides which code path is better. They share one mechanism, splitting traffic, and nothing else. Every A/B test needs a flag underneath it, but the large majority of flags are not experiments and never will be. If you are shipping features and want to release them safely, you need flags. If you are settling arguments about which version wins, you need an experimentation platform, which is a much larger and much more expensive purchase. Most teams that buy the second one only ever needed the first.
That distinction is worth a few hundred dollars a month at small scale and six figures a year at large scale, so it is worth getting right before the sales calls start.
Is a feature flag the same as an A/B test?
No. A feature flag is a conditional in your code whose value is controlled from outside the deploy. It answers "should this user see the new checkout?" An A/B test is a statistical procedure: assign users randomly, hold everything else constant, collect a metric, and decide whether the difference you observed is real or noise. It answers "did the new checkout make us more money?"
The confusion is understandable, because both start the same way. Both split traffic. Both need stable assignment so a user gets the same experience on every request. Both need a way to target segments. If you look only at the assignment layer, they are the same thing, and that is exactly what vendors selling experimentation platforms would like you to conclude.
The difference shows up in what happens after assignment. A flag is done once the user is routed. An experiment is barely started: it needs a pre-registered metric, a sample size calculated before launch, a fixed run length, a significance test, and someone willing to accept a result they did not want.
| Feature flag | A/B test | |
|---|---|---|
| Purpose | Control exposure | Measure impact |
| Success looks like | Nothing broke, everyone is on the new version | A defensible number, even if it says no |
| Typical lifespan | Days to weeks, then deleted | Two to four weeks, fixed in advance |
| Who decides | The engineer shipping the change | Whoever owns the metric |
| Needs statistics | No | Yes, and the details matter |
| Needs a clean event pipeline | No | Absolutely, this is the hard part |
| Cost if you get it wrong | An incident, resolved in seconds | A confident decision based on nothing |
What a feature flag does that an A/B test does not
Most of the value teams get from flags has nothing to do with measurement.
It separates deploy from release. Code goes to production dark, merged and running, visible to nobody. The release becomes a separate decision made later, by a different person, without a deploy. That single change is what makes trunk-based development workable at scale, and it removes the Friday-afternoon deploy freeze most teams pretend they do not have.
It gives you an off switch. When something goes wrong at 2am, the recovery path is flipping a boolean rather than reverting a commit, waiting for CI and rolling a deploy. That gap is the difference between a four minute incident and a fifty minute one. A feature flag kill switch is worth having even in a codebase that never runs a single experiment.
It handles gradual exposure. One percent, then five, then twenty five, watching error rates the whole way. This is not an experiment. Nobody is computing significance. You are looking for a spike in your existing dashboards before the blast radius gets large. The mechanics are in our guide to running a gradual rollout.
It gates entitlements. Which plan sees which features, which tenant gets the beta, which internal account can see the unfinished thing. These flags are permanent infrastructure and they will still be there in three years.
Add those up and you have the honest picture of a mature flag setup: mostly release safety, mostly permanent configuration, and a small minority of flags that ever carry a hypothesis.
What an A/B test does that a feature flag does not
An experiment is a claim about causality, and holding that claim up requires machinery a flag system does not have.
You need metric definitions that everyone agrees on before the test starts, not after the numbers arrive. You need a sample size calculated up front, because deciding when to stop after you have seen the data is how teams manufacture results that do not replicate. You need guardrail metrics, so a variant that lifts clicks while quietly increasing refunds gets caught. You need to handle multiple comparisons, because testing six variants against one control and celebrating the winner is a coin-flip exercise dressed up in a dashboard.
And you need the underlying data to be trustworthy, which is the requirement nobody puts on the shopping list. An experiment result is only as good as the pipeline feeding it. A metric that silently stopped updating on day three will happily report a flat result for the remaining eleven days, and nothing in the experimentation tool will tell you. Keeping an eye on freshness, volume and schema drift in the tables underneath is a separate discipline from experimentation, and skipping it is how organizations end up making decisions from a chart that stopped being true a week ago.
That is real work, and platforms like Optimizely and Statsig have built serious products around it. If your company runs experiments as a discipline, with results that get challenged in review, that maturity is exactly what you are buying and it is worth paying for.
Can you do A/B testing with feature flags?
Yes, and for most teams it is the right starting point. A flag with a fifty-fifty percentage rollout, stable user bucketing and an exposure event sent to whatever analytics tool you already use will get you a legitimate A/B test. The assignment layer is genuinely the same. What you are supplying yourself is the analysis: the sample size, the run length, and the significance calculation.
This works well when tests are occasional and the effects you are looking for are large. It works badly when you are running twenty concurrent experiments, need sequential testing so you can peek at results without inflating false positives, or have to defend a two percent lift to a finance team. At that point you have outgrown the do-it-yourself version, and that is a good problem. Our walkthrough of A/B testing with feature flags covers the practical setup, including the parts that are easy to get wrong.
The honest failure mode of the do-it-yourself approach is not statistical, it is social. Without a tool enforcing the process, someone will stop a test early because the number looked good on Thursday. Enforcement is a real feature.
Do I need an A/B testing tool if I already have feature flags?
Count the experiments you actually ran in the last twelve months, not the ones you meant to run. If the answer is fewer than one a month, and none of them changed a decision that was already made, an experimentation platform is not your bottleneck. Buying one will not create a testing culture; it will create an unused license and a quarterly conversation about adoption.
If the answer is several a month, with real disagreement about the results, then the platform pays for itself in decisions that stop being arguments. The tell is whether anyone has ever changed their mind because of a test result. If nobody has, the missing ingredient is not software.
There is a middle position that suits a lot of US SaaS teams: flags everywhere for release safety, plus a handful of properly run tests per year on the things that genuinely move revenue, analyzed in the warehouse you already have. That gets you most of the value at a fraction of the cost, and it leaves the door open to buying a real experimentation platform once the volume justifies it.
The pricing consequence nobody mentions
This is the part that turns a definitional question into a budget question.
Flag tools and experimentation platforms are priced differently because they are sold to different buyers. Flag platforms tend to publish a number: ConfigCat lists $110 for Pro and $325 for Smart, Flagsmith lists $40 and $250 for its two paid tiers, and both include seats you do not have to count. Experimentation platforms tend not to publish anything. Optimizely redirects its pricing page to a plans page that says every plan is individually packaged, and Harness does not publish a figure for the product it acquired from Split either. Statsig is the notable exception, with $150 a month for Pro including five million events.
So the question "is this a flag or an experiment?" quietly decides whether you buy from a price list or enter a sales cycle. Teams that answer it carelessly end up licensing a statistics engine, a visual editor and a personalization layer in order to switch a checkout page on for ten percent of users. We put the actual figures side by side on our Optimizely feature flags page, and the wider pattern of what vendors choose to meter is in feature flag pricing models.
What is the difference between a feature flag and an experiment?
A feature flag is a mechanism. An experiment is a method that uses that mechanism. Every experiment runs on a flag, but calling a flag an experiment is like calling a stopwatch a study. The flag decides who gets the treatment. The experiment specifies the hypothesis, the metric, the sample size and the stopping rule, and then commits to the answer in advance. If nobody wrote down what would count as a failure before launch, what you ran was a rollout with a chart attached.
Three mistakes that cost real money
Calling a rollout an experiment. Shipping to ten percent, seeing conversion tick up, and rolling out to everyone is not a test. Traffic at ten percent is rarely comparable to the other ninety, the sample is usually far too small, and the decision was already made. This is the most common form of measurement theater, and it is worse than not measuring, because it produces false confidence.
Buying experimentation to get flags. The flags inside an experimentation platform are usually good. They are also attached to a price shaped by the statistics engine. If your requirement is release safety, compare against tools that only do release safety before you sign anything.
Leaving experiment flags in the code. An experiment ends, a winner is picked, and the flag stays for two years because deleting it is nobody's job. Experimentation platforms are enthusiastic about creating flags and indifferent about retiring them. Set an expiry date at creation time, which is one of the habits in our feature flag best practices.
How to decide in ten minutes
Write down the last five changes your team shipped that mattered. For each one, answer two questions: did we need to control who saw it, and did we need to prove it worked? Almost every change will answer yes to the first. The proportion answering yes to the second is your real experimentation rate, and it is usually far lower than the roadmap implies.
If that number is low, buy flags, keep them flat-priced so growth does not punish you for instrumenting your own product, and run the occasional test on the flags you already ship with. If it is high, buy an experimentation platform and accept the sales cycle, because the statistics are the product and you will use them.
Either way, wrap the vendor behind your own function on day one. It costs an afternoon and it means the answer to "what would it cost to change our mind about this?" is a file rather than a quarter. That mechanism, and what it saves, is laid out in feature flag migration. If you are still shortlisting, the current field with verified July 2026 prices is in best feature flag tools, and the A/B testing software page shows what testing looks like when it runs on the flags you already have.