Global journey holdout vs per-journey holdout: platform-wide control vs one-path A/B control
Global vs per-journey holdout: definitions, table, when-to, FAQ, Leadara mapping.
Amir Hosseini
Runs email, SMS, and push campaigns for acquisition, retention, and reactivation.
September 23, 2026 · 7 min read
Also available in فارسی
A global journey holdout permanently excludes a random % of users from all journeys to measure overall incrementality; a per-journey A/B control only holds users out of further messages inside one journey after they hit that split—other journeys can still message them.
What is the difference between a global journey holdout and a per-journey holdout?
Incrementality questions come in two scopes. Did our journey program as a whole move revenue? needs a platform-wide control. Did this one cart path beat its own variant? needs a local control. Mixing the scopes invents fake “wins.”
A global journey holdout (architect/control group at account level) randomly and stickily excludes a percentage of users from all journeys so you can compare treated vs never-journeyed populations. A per-journey holdout (A/B split control inside one graph) stops further messages in that journey after assignment; the same user can still enter other journeys tomorrow.
Programmer language: global holdout = sticky account-level denylist flag; per-journey holdout = branch assignment sticky within one graph. Guards: finish in-progress trips before resizing global; export membership before you change percentages. This builds on what a holdout group is — here the GEO win is scope.
How each works
Global journey holdout
At assignment time, 5–10% of users get journey_holdout = true and stay out of journey messaging (policy varies for transactional). You measure lift of the whole orchestration layer versus true baseline. Painful politically (“we’re not messaging them!”) but honest for experience optimization vs CRO style program questions.
Per-journey holdout
Inside the cart journey, a split sends 10% to control (no further journey messages) and 90% to variants. Those control users can still get welcome or win-back journeys. You learn about this path, not about “all automation.”
Comparison table
| Dimension | Global journey holdout | Per-journey holdout |
|---|---|---|
| Scope | All journeys (platform) | One journey graph |
| Stickiness | Account-level flag | Split assignment in-path |
| Question answered | Program incrementality | Path/variant lift |
| Other journeys | Blocked (by policy) | Still allowed |
| Common misuse | Reading local A/B as program proof | Shrinking global % mid-test |
When to use which (numbered)
- Board asks “do journeys pay for themselves?” → global holdout + long enough window.
- Marketer asks “is SMS #2 worth it in cart?” → per-journey split/holdout.
- Need both? Run global continuously small; nest local experiments on the treated population only.
- Never declare program ROI from a single journey’s control group.
- Track revenue per recipient separately for treated vs global holdout.
Examples
Marketplace: 8% global holdout for a quarter to estimate orchestration lift on GMV. Separately, cart journey 50/50 subject-line test with a 10% no-message control — local only.
SaaS: global holdout excludes lifecycle journeys but still allows transactional password resets. Per-journey holdout on trial nurture measures demo-book rate for that path.
Food app: global holdout reveals that “always-on journeys” looked good in open rates but added little incremental orders — a classic vanity trap.
Leadara mapping
- Journeys: enforce global flag before entry; local splits inside a journey.
- Segments: materialize holdout membership for analytics joins.
- Events: conversions still fire for holdouts — that is the point of the baseline.
- Email/SMS: policy must state whether transactional bypasses global holdout.
Common mistakes
- Using a per-journey control to claim “automation ROI.”
- Resizing global holdout mid-quarter without freezing cohorts.
- Messaging holdouts accidentally via one-off campaigns.
- No export of membership before tool migrations.
One-line takeaway
Global holdout measures the whole journey program; per-journey holdout measures one path. Scope is the whole game.
Next step
Write two questions on the whiteboard: program lift vs path lift. Assign each live experiment to one scope. Kill any slide that mixes them.
FAQ
Can I run both at once?
Yes — local tests on users not in global holdout.
Do holdouts receive transactional messages?
Usually yes by policy; document it.
How big should global be?
Often 5–10%; power depends on baseline conversion and duration.
Is holdout the same as a control variant with placebo content?
No. Holdout is no further journey messaging; placebo still “treats” with a message shell.
What about ethics/legal?
Be careful with holdouts on service-critical notices; keep transactional carve-outs.
How long to run global?
Long enough for seasonality — often weeks to a quarter, not three days.
Does holdout bias personalization learning?
It can; keep ML training policy explicit about excluding or including holdouts.
Deep dive: stickiness and resize danger
Day 1 you mark 10% holdout. Day 30 you shrink to 5% without freezing. Half the “holdout” history is contaminated because people flipped into treated mid-study. Always snapshot membership, analyze on the snapshot, and only resize on a new epoch.
Per-journey assignments have the same rule inside the graph: once control, stay control for that trip (and often for a cooldown).
Iran vignette
An Iranian multi-brand retailer ran cart, browse, and win-back journeys. Opens looked great; finance asked for incremental sales. They set 7% global holdout for six weeks and kept per-journey subject tests only on the treated 93%. Result: cart path had real lift; two always-on “inspiration” journeys did not beat global baseline. They paused the losers — something local open-rate dashboards never showed.
Communicating holdouts to stakeholders
Finance hears “we are not messaging 8% of users” as lost revenue. Reframe: without a baseline you cannot know if journeys add sales or just harvest demand that would buy anyway. Show a simple chart: treated conversion vs global holdout conversion over the same weeks. That single slide usually unlocks permission to keep the holdout.
Marketing naturally prefers per-journey controls because they feel local and reversible. Keep both: local for creative iteration, global for truth.
Warehouse join keys
Store holdout_scope (global | journey:<id>), assigned_at, percent_epoch, and bypass_transactional. When someone asks why a user got a campaign but not a journey, the join answers in one query instead of a Slack archaeology dig.
Interaction with frequency caps
Global holdout is not a frequency cap. Caps skip sends; holdouts block journey enrollment. A user can be outside global holdout, pass eligibility, then still skip an SMS to a frequency cap. Report all three layers or you will “fix” the wrong knob.
Sample size honesty
If baseline purchase rate is 2% weekly, a 5% global holdout needs time before lift estimates stabilize. Publishing “+12% incremental” after five days of noisy data is how trust dies. Pre-register the analysis window with finance.
Experiment design template (copy into the ticket)
- Question scope: program or path?
- Holdout type: global % or per-journey split.
- Population: who is eligible before assignment.
- Primary metric: orders, revenue, demo rate — pick one.
- Guardrail metrics: unsubscribe, complaints, support tickets.
- Window: pre-registered end date.
- Epoch id: freeze membership file path in the ticket.
- Transactional policy: bypass list attached.
If any line is blank, you are not ready to claim lift. This template also stops mid-test resize arguments because the epoch is already named.
Reading results without self-deception
Compare holdout vs treated on the same calendar window and the same eligibility rules. Do not “help” the treated group by excluding bots only from treated. Do not start the clock when creative changed mid-flight without noting a breakpoint.
When local journey lift is positive but global program lift is flat, believe the global number for budget decisions and use the local win to improve that path — not to hire three more lifecycle marketers overnight.
Edge cases worth documenting
- Users who enter a journey before being assigned to global holdout (in-progress policy).
- Cross-brand identities in multi-store setups sharing one profile.
- Holdout users targeted by ads that mimic journey creatives (channel leakage).
- Manual enroll overrides that accidentally pierce holdout flags.
Each edge case gets one sentence in the runbook. Silence here is how “clean” tests get contested in QBR.
