How to roll out AI features safely with feature flags

The short answer: put four flags around every AI feature, not one. A kill switch that falls back to the non-AI path, a model selector so you can change provider or version without a deploy, a percentage rollout keyed to a stable user id, and a cost ceiling that degrades to a cheaper model or a queue when spend crosses a threshold. AI features fail differently from normal features: they degrade in quality instead of throwing errors, their cost per request is unbounded, and the vendor can change the model underneath you. A boolean on and off flag does not cover any of those three.

This is written for teams shipping LLM features into production software, not for research work. Everything below assumes the feature is in front of paying customers.

Why AI features need different flags

The normal case for a feature flag is binary: the new checkout either works or it does not, and your error rate tells you which. Three properties break that model.

Failure is gradual, not binary. A model that has started returning subtly worse answers throws no exceptions and fires no alarms. Your latency is fine, your 500 rate is fine, and the feature is quietly getting worse. Detection depends on quality signals you had to build in advance: thumbs-down rates, human review samples, task completion, escalation to support. If you have no quality signal, you have no way to know you should pull the flag.

Cost per request is unbounded. A conventional endpoint costs roughly the same every time. An LLM call costs whatever the input and output tokens add up to, and a user who pastes a fifty-page document costs a hundred times what the median user costs. Traffic doubling is a capacity problem; a change in how people use the feature is a budget problem, and it can arrive without any change on your side.

The dependency changes without you. Providers release new model versions, deprecate old ones, adjust defaults and change safety behavior. You can pin a version, and you should, but pinned versions get retired on the provider's schedule. That means the ability to switch model at runtime is not a nice-to-have, it is your response to a dependency you do not control.

Flag one: the kill switch, and what it falls back to

Every AI feature needs a switch that turns it off in seconds without a deploy. That much is standard, and our page on the kill switch pattern covers the general mechanics.

The AI-specific part is the fallback. Turning off a normal feature reveals the old feature. Turning off an AI feature often reveals nothing, because there was no old feature; the product was built assuming the model answers. So decide the degraded state before you launch and build it at the same time as the happy path. Usually one of: the pre-AI manual workflow, a cached or templated response, a smaller and cheaper model, or an honest message that the assistant is unavailable with a clear route to a human.

Test the fallback the way you would test a backup restore. A kill switch nobody has pulled in staging is a hypothesis. Pull it deliberately in production during a quiet hour, watch what customers actually see, and fix what is ugly. The moment you need it, you will be under pressure and it will be the wrong time to discover the degraded path renders a blank panel.

Flag two: the model selector

Make the model a flag value, not a constant. A string flag returning a model identifier, read at request time, is the single highest-leverage flag in an AI feature. It gives you four things at once: switch provider during an outage, roll a new model version to 5% of traffic, run a cheaper model for free-tier users, and revert instantly when a new version turns out worse for your specific task.

// One string flag, four capabilities.
const model = await flags.getString('assistant.model', 'claude-sonnet-5', {
  targetingKey: user.id,
  plan: user.plan,
  workspace: user.workspaceId,
});

const prompt = await flags.getString('assistant.prompt_version', 'v3', ctx);

const answer = await llm.complete({ model, prompt: prompts[prompt], input });

Flag the prompt version alongside the model. Prompts change behavior as much as model choice does, and a prompt change is a release even though it does not look like code. Teams that flag the model but hardcode the prompt end up redeploying to fix wording, which defeats the point.

One caution: keep the set of valid values small and validated. A string flag that can hold anything is an outage waiting for a typo. Constrain it to a known list in code and fall back to the default when the value is unrecognized.

Flag three: the percentage rollout, keyed properly

Roll AI features out gradually for a reason that is different from the usual one. With a normal feature you are watching for errors, and errors show up in minutes. With an AI feature you are watching for quality, and quality signals are slow, noisy and often human. That argues for holding each step longer than you would for a conventional release, not shorter.

A workable ladder: internal users, then 1%, then 5%, then 25%, then half, then everyone, holding each step until you have enough quality signal to mean something rather than for a fixed number of hours. Our gradual rollout guide covers the general mechanics and what to watch at each step.

Two AI-specific rules. First, bucket on a stable identifier, and for B2B products bucket on the workspace or account rather than the individual, so a whole team sees consistent behavior instead of two colleagues getting different answers to the same question and filing a bug about it. Second, make the assignment sticky. Nothing erodes trust in an assistant faster than it being noticeably good on Tuesday and noticeably different on Wednesday because the rollout percentage moved and rebucketed the user.

The major platforms have converged on this. AWS added enhanced targeting during rollout in March 2026 specifically so values stay sticky to an entity for the life of a gradual rollout, and its managed experimentation tools, generally available in 2026, explicitly cover model selection and prompt experiments alongside UI changes. That is a reasonable signal about where this pattern is going.

Flag four: the cost ceiling

This is the one teams add after the first surprising invoice rather than before, and it is the one most specific to AI.

Track spend per feature and per tenant, and wire a flag to the threshold. When a tenant crosses its daily budget, the flag flips them to a cheaper model, a smaller context window, a queue, or a message explaining the limit. The flag makes the response instant and reversible, which matters because the alternative is a human noticing a dashboard on a weekend.

Set thresholds per tenant rather than globally where you can. A single customer running an automated integration against your assistant can consume more than your entire remaining user base, and a global ceiling means their behavior degrades everyone else's experience. Per-tenant limits contain the blast radius to the account causing it, which is also the account you want to have a pricing conversation with.

Rate limits belong in the same family. A flag-controlled requests-per-minute value per plan tier lets you tighten the tap during an incident without shipping code, and loosen it again when the provider recovers.

Flags for AI agents specifically

If your feature is an agent that calls tools rather than a single completion, the surface area grows and so does the flag list.

Flag each tool independently. An agent with access to search, code execution, email and a database should have four flags, not one, so a misbehaving tool can be removed without disabling the agent. Flag the maximum step count too, because runaway loops are the characteristic agent failure and a flag is how you cap them without a deploy.

Gate the genuinely irreversible actions behind their own flags with tighter targeting than the agent itself: sending external email, moving money, writing to production data, deleting anything. Those should be off by default, on for a named allowlist, and separately killable. The agent being enabled and the agent being allowed to send email on your behalf are two different risk decisions and they should not share a toggle.

Flags control what your agent is permitted to do; they do not validate what comes back or what goes in. An agent that reads untrusted content, a support ticket, a scraped page, a user-supplied document, can be steered by instructions hidden in that content, and no rollout percentage protects you from it. That is a separate control layer: you want something inspecting tool calls and enforcing data boundaries so a prompt injection cannot turn a read-only assistant into one that exfiltrates a customer record. Treat the flag as the switch and that layer as the seatbelt. You need both, and they solve different problems.

What to measure before you widen the rollout

Instrument these before the first percentage step, because retrofitting them mid-rollout means comparing against no baseline.

• Quality: explicit feedback rate, regeneration rate, and a small human-reviewed sample. Regeneration rate is the most honest cheap signal you have, since a user asking again is telling you the first answer failed.
• Cost per request and per tenant, broken down by flag variant, so a model change shows up as a cost change you can attribute.
• Latency at p95 and p99, not the mean. Model latency has a long tail, and the tail is what users complain about.
• Fallback and error rate, including provider timeouts, rate limits and refusals, which are normal operating conditions rather than exceptions.
• Escalation to human support, which is the business metric that decides whether the feature is working at all.

Tag every one of these with the flag variant that produced it. Without variant tagging you can see that something changed but not which decision changed it, and with several flags moving at once that is a genuinely hard forensic problem.

The starting configuration

If you are shipping an AI feature next week, this is a reasonable day-one setup: one boolean master switch with a tested fallback, one string model flag, one string prompt version flag, one percentage rollout bucketed on workspace id, one numeric daily cost ceiling per tenant, and one boolean per irreversible tool if there is an agent involved. Six flags, all owned by name, all with a removal or permanence decision recorded when they are created.

That last part matters more than it sounds. AI features generate flags quickly because there are so many knobs worth controlling, and flags that outlive their purpose are the main long-run cost of any flag program. Decide at creation time whether each one is temporary or permanent infrastructure, and write it down. The feature flag best practices post covers the naming and ownership conventions that keep this from turning into fifty stale conditionals a year from now.

All of it runs on ordinary feature flag machinery. There is no AI-specific flag product you need. What is specific is the discipline: string flags rather than booleans, sticky bucketing, a fallback you have actually tested, and a cost ceiling wired up before the invoice teaches you to want one. A feature flags platform with fast propagation, consistent bucketing across every language and a kill switch anyone on call can pull is all the tooling this requires.