Holdout Groups & Lift
A holdout group is a slice of your target audience that is deliberately not sent a message. After the send, you compare conversion rates between the subscribers who received the message (the treated group) and those who were held out — the gap between the two rates is the send's incremental lift: what the message actually caused, rather than what would have happened anyway.
Holdouts answer a different question than A/B variants. Variants compare which message performs better; a holdout measures whether sending at all made a difference. The two combine freely — see A/B Variant Sending.
Holdout groups are gated by the same plan feature as A/B variant sending. On plans without that feature, the API returns the same 403 PLAN_FEATURE_UNAVAILABLE error as a variant send.
1. Setting a holdout
In the dashboard Composer
- Open the Composer and fill in your message as usual.
- Below the content card, check Hold out a control group — the label reminds you what it does: a percentage of the audience receives nothing, for lift measurement.
- A Holdout % slider appears, defaulting to 5%. The dashboard slider ranges from 1% to 50%; the readout below it shows the split live (for example Held out: 10% / Receiving: 90%).
- Send as usual. The held-out share is applied when the job fans out.
A holdout of 5–20% is typical: large enough to measure against, small enough not to cost you much reach.
Via the API
Add holdout_percentage to the create request. The API accepts any value from 0 (no holdout, the default) to 99; values outside that range are rejected as a validation error.
POST /api/v1/notifications
Authorization: Bearer <token>
Content-Type: application/json
{
"channel_ids": ["chan-uuid"],
"target": { "type": "all" },
"content": { "title": "Flash sale", "body": "Ends tonight" },
"holdout_percentage": 10
}
Holdouts work on standard sends and on A/B variant sends. When combined with variants, the variant percentages still sum to 100 and split only the treated remainder.
On campaigns
Campaigns accept a campaign-level holdout_percentage on create and update. When the campaign is released, the same holdout share is stamped onto every message in the campaign, and the campaign detail page shows the configured Holdout percentage.
Note that holdout membership is re-drawn per send (see below), so within a multi-message campaign a subscriber may be held out of one message and treated on another. Lift is read per message, using each message's job ID.
2. How holdout assignment works
Assignment is deterministic, per subscriber:
bucket = hash(subscriber_id + job_id) % 100
bucket < holdout_percentage → held out
This is deliberately different from variant assignment in two ways:
- Per subscriber, not per device. Lift is a behavioral comparison between people. A subscriber with two devices is either fully held out or fully treated — never half of each.
- Seeded with the job ID. Membership re-randomizes on every send, so no subscriber is permanently starved of messages the way a device permanently sticks to one variant.
Devices that are not linked to an identified subscriber are always treated — an anonymous device cannot contribute to a subscriber-level comparison.
Because the hash of the same (subscriber, job) pair always produces the same result, the holdout set is stable across retries and redeliveries: re-processing a job reproduces exactly the same membership.
3. What happens to held-out subscribers
- Their membership is recorded before any delivery starts, so the control group exists even if the send is interrupted and retried.
- They receive nothing: no delivery is attempted, no delivery records are produced, and held-out subscribers consume no send quota.
- A job whose entire (small) audience happens to land in the holdout completes successfully with zero sends — it is not a failure.
4. Reading the lift
Prerequisite: conversion events
Lift is measured in conversions, so you need at least one conversion event defined before the readout means anything. Configure conversion events on the Insights page — each definition names a tracked event (optionally with a revenue value and an attribution window). Without any definitions, the lift readout reports conversion_configured: false and both sides are vacuously zero. See Server-Side Event Tracking for how to send the underlying events.
In the dashboard
Open History, click the job, and scroll to the Holdout Lift section (shown only for jobs that carried a holdout). It displays:
- The lift in percentage points, with the measurement window (for example Lift: +5.0 points over a 7-day window). Negative lift is shown as-is — it means the held-out group converted more.
- A Received vs Held out table with each group's subscriber count, converters, conversion rate, and attributed revenue.
Via the API
GET /api/v1/insights/lift?job_id=<job-uuid>
{
"job_id": "job-uuid",
"treated": { "size": 900, "converters": 90, "conversion_rate": 0.10, "revenue_cents": 45000 },
"holdout": { "size": 100, "converters": 5, "conversion_rate": 0.05, "revenue_cents": 2500 },
"lift_percentage_points": 5,
"window_days": 7,
"conversion_configured": true
}
| Field | Description |
|---|---|
treated / holdout | Per-group counts: size (subscribers), converters, conversion_rate (converters ÷ size), revenue_cents (attributed conversion revenue). |
lift_percentage_points | Treated conversion rate minus holdout conversion rate, in percentage points. May be negative. |
window_days | The conversion window applied to both groups. |
conversion_configured | false when no conversion events are defined — the counts are meaningless until you configure one. |
How the numbers are computed
- Each treated subscriber is anchored at their first delivered touch for the job; each held-out subscriber is anchored at the moment of assignment.
- A subscriber counts as converted when they have at least one conversion event within the window of their anchor.
- The window is the widest attribution window across your configured conversion events, defaulting to 7 days when none specify one.
Small groups make noisy lift. A 5% holdout on a 500-subscriber audience is a control group of ~25 people — judge the rates alongside the group sizes before drawing conclusions.