What is a holdout (control) group in marketing? How to measure true lift

What is a holdout group in marketing automation and how is it different from an A/B test?

A holdout (control) group is a slice of eligible people you deliberately keep out of a marketing action — for example a cart journey, win-back, or discount offer — so you can see whether conversion lift came from the message or from natural buying. Without that slice, every campaign that “produces sales” looks successful, even when those sales would have happened with no email or SMS.

One-line answer: an A/B test usually compares two treatments (subject line A vs B). A holdout creates a no-treatment group so you can measure incremental lift. Both are measurement tools; they answer different questions.

If your team still celebrates opens and CTR alone, ask before you scale sends: do we know this journey added sales — or only reached people who would have bought anyway? For decisioning and where holdouts sit in next-best-action, see: What is AI decisioning in marketing?.

A practical definition for eCRM teams

Operationally, a holdout means:

1. Select eligible people with one rule (the same journey-entry segment).
2. Exclude a stable random share from the action — not “whoever entered late.”
3. Compare treatment vs holdout conversion/revenue in a fixed window.
4. Judge success by the gap, not by the treatment rate alone.

Nearby terms: control group, incremental lift, ghost control, program holdout. Same idea: no message, on purpose, for comparison.

A note for Iranian ecommerce teams: many shops treat holdout as “lost sales” and delete it. The real cost is burning margin on unnecessary discounts and false confidence in campaigns — not the 10% who missed one week of promo touches.

Sizing: what holdout percentage should you use?

There is no magic fixed share; it depends on volume, conversion variance, and opportunity cost. A common marketing-automation pattern:

Sizing principles that matter more than the number:

• Stable randomness: the same person should not flip between treatment and control inside the window.
• Lock the window up front: 7 or 14 days; do not move it after seeing results.
• Enough volume: if the holdout yields only a handful of conversions, the result is noise — widen the window or share thoughtfully, not by gut feel.
• One claim per test: do not change discount, channel, and send time at once.

If the database is small, do not drop holdout entirely — lengthen the window or hold out only the riskiest branch (for example deep-discount SMS).

Holdout vs A/B vs global program holdout

Table takeaway: first prove the action is worth it (journey holdout), then optimize the version (A/B), and occasionally check the whole program with a global holdout. A/B without a no-treatment control only tells you which spam opens a bit better.

When to skip or shrink holdout (transactional caveat)

Holdout is for measuring optional / promotional actions — not an excuse to withhold necessary messages.

Usually skip or keep holdout tiny when:

1. Transactional messages — order receipts, OTPs, shipping status, legal notices. A “no message” control here breaks the experience; it is not science.
2. Product-mandatory paths — account activation that cannot complete without the email; deliberate silence is meaningless.
3. Very low volume + high customer risk — e.g. critical subscription-expiry reminders with tiny counts; operational certainty first, then experiments.
4. Purely technical fixes with no lift claim — repairing a broken link does not need a sales holdout.

Clear boundary: a discounted cart reminder is promo and wants a holdout; “your order was placed” is transactional and does not. If one journey mixes both, split them — then hold out only the promo branch. Keep the channel frame nearby: Transactional vs promotional email and SMS (and see lifecycle / journey posts in Related reading).

Which metrics show true lift?

Golden rule: if uplift vs holdout is weak, do not send more — change the message, offer, or path priority.

How to approximate this in Leadara with events, segments, and journeys

Leadara runs on events, segments, journeys, email, and SMS. You do not need a separate “holdout engine” feature to run a real control:

1. Send clean events for eligibility and conversion: e.g. cart_updated, order_completed, channel consent.
2. Build the entry segment as the people who would enter the journey today.
3. Build a stable random holdout segment — e.g. 10% from a stable ID hash — and suppress them from journey entry or give a no-send branch.
4. On the treatment journey, add conversion exits and frequency caps.
5. Weekly view: treatment vs holdout conversion and RPU in the locked window.
6. Keep promo separate from transactional so holdout never blocks necessary messages.

If several journeys run in parallel, define holdouts per path; otherwise someone in cart holdout may still get Friday blast and break the control. For behavioral path vs blast framing: Journey vs campaign in marketing automation.

Worked scenario: fashion ecommerce

Imagine “Nora” runs a cart journey at 15% off, a welcome with the same code, and a Friday SMS — with no holdout.

Diagnosis

Fix with holdout

• Stable 10% holdout on discounted cart; compare 7-day conversion.
• Remove Champions and recent buyers from deep Friday discounts.
• Keep welcome from stacking an extra cart code.
• SMS only with consent, caps, and high-value targeting.
• Success = uplift vs holdout + STOP control — not send volume.

Common mistakes

• Dropping holdout because it “loses sales”
• Changing percentage or window after a weak result
• Substituting subject-line A/B for a no-treatment control
• Holding out transactional sends
• One shared holdout across overlapping journeys without mutual suppress
• Optimizing CTR instead of incremental revenue and margin

Checklist for this month

1. Pick one discounted journey and build a stable 10% holdout
2. Write and lock a 7- or 14-day window
3. Separate transactional from promo
4. Cut welcome / cart / blast overlap
5. Dashboard uplift, RPU, STOP
6. Scale or run creative A/B only if uplift exists

FAQ

What exactly is a holdout group in marketing?

A group of eligible people who deliberately do not receive the automated action so you can measure the message’s incremental effect against natural baseline.

How is a holdout different from an A/B test?

A/B compares two (or more) treatments. A holdout compares treatment to no treatment so true lift is visible.

What holdout percentage is enough?

For high-risk paths, 10–20% is a common start. More important: stable randomness, a fixed window, and enough conversions.

Do SMS journeys need a holdout too?

Yes — especially with promo offers or urgency. SMS has high relationship cost; without a control, STOP and fake sales can rise together.

When do you need a global program holdout?

When you want to know how much incremental sales the whole promo automation stack creates — not just one journey — usually in a seasonal review or before a large budget.

Is a holdout required for welcome email?

If welcome is education/activation with no sales-lift claim, product experience comes first. If welcome sells the same deep discount, a holdout or controlled sample is useful.

How do you know the result is noise?

If either arm has very few conversions in the window, or you changed the percentage mid-test, do not trust the number — fix the design and measure again.

How does Leadara support holdouts conceptually?

With events, segments, email/SMS journeys, suppressions, and exits. You implement control as a stable random segment plus metric comparison; you do not need a separate branded holdout product.

Bottom line and next step

A holdout is the courage not to message part of the eligible audience so you can see where true lift lives. It is not an A/B test, it must not touch transactional sends, and it is critical on discounted paths. This month, measure one high-risk journey with a stable 10% control and a locked window; if uplift is missing, do not scale — fix the message and priority first.

Related reading

• What is AI decisioning in marketing? Next-best-action explained
• Journey vs campaign in marketing automation: what’s the difference?
• What is lifecycle marketing? Onboarding, retention, and win-back