A/B Variant Sending

Growth plan required

A/B variant sending is available on the Growth plan and above (the ab_testing feature). You will receive a 403 PLAN_FEATURE_UNAVAILABLE error if you attempt to use variants on the Free or Starter plan. To upgrade your plan, visit your billing settings.

A/B variant sending lets you deliver two different versions of a push notification to separate segments of your audience and compare per-variant delivery counts. You define a split — for example, 60% of devices see Variant A and 40% see Variant B — and the platform handles the assignment automatically.

This guide covers how variant assignment works, current limitations, full API examples, how to read per-variant results, and a dashboard walkthrough.

1. Overview

When you create a notification job with variants, Hober splits your target audience across 2–4 content variants according to the percentages you specify (they must sum to 100). Each device in the target receives exactly one variant. After the job runs, the notification detail page shows a per-variant breakdown of delivered, failed, and pending counts.

What A/B testing in Hober does (Sprint 11 scope):

Sends 2–4 content variants to different device segments in a single notification job
Tracks delivery counts, failures, and pending sends separately per variant
Tracks outcomes per variant: taps, attributed conversions, and refund-netted revenue per currency — using the same revenue attribution semantics as every other surface, so the breakdown answers "which variant made more money", not just "which got delivered"
Uses a deterministic, hash-based assignment so the split is stable and reproducible
Optionally runs as an auto-winner experiment: send the variants to a test slice first, evaluate a metric after a window you choose, and roll the winning content out to the rest of the audience automatically — see Auto-Winner Experiments
Supports per-message variants (and auto-winner) on campaign messages

What A/B testing does not do in this release:

No statistical significance calculator — the auto-winner gate is a reliability heuristic (minimum sample sizes), not a significance test
No multi-armed bandits — traffic split percentages are fixed for the life of the job

Holdout groups — withholding the send from a slice of the audience to measure incremental lift — are a separate, complementary feature. See Holdout Groups below.

2. How Variant Assignment Works

Variant assignment is hash-based and deterministic per device ID. The same device always receives the same variant for any given split percentage, regardless of when or how many times you query the assignment.

Assignment algorithm

For each device in the target audience, Hober computes:

bucket = hash(device_id) % 100

The bucket is an integer between 0 and 99 (inclusive). The device is then assigned based on the split you configured:

The variants' percentages partition the bucket space 0–99 in order: with a 60/40 split, buckets 0–59 receive Variant A and 60–99 receive Variant B; with a 50/30/20 three-way split, buckets 0–49 receive A, 50–79 receive B, and 80–99 receive C.

Stability of the split

Because the hash input is the device ID alone — with no job ID or timestamp component — the same 60% of devices always fall into Variant A for any job using a 60/40 split. The assignment does not shift between jobs.

This means:

Reproducible: re-running the same job delivers the same variant to the same devices.
Consistent: a device's assigned variant is predictable and auditable.
Not cross-job randomized: if you want a different 60% to see Variant A on a follow-up job, you must change the split percentages.

3. Limitations

Limitation	Detail
Percentages must sum to 100	The `percentage` fields of both variants must add up to exactly `100`.
No significance calculator	The auto-winner gate checks minimum sample sizes, not statistical significance.
Variants are immutable	After a job is created, its variant content and percentages cannot be changed. Create a new job to adjust the split or content.
Cannot mix with `content` or `templateId`	A job that uses `variants` must not also include a top-level `content` object or a `templateId`. These are mutually exclusive.

4. API Example

Create a notification with variants

POST /api/v1/notifications
Authorization: Bearer <token>
Content-Type: application/json

{
  "channelIds": ["chan-uuid"],
  "target": { "type": "all" },
  "variants": [
    {
      "label": "A",
      "content": { "title": "Hello!", "body": "Variant A body" },
      "percentage": 60
    },
    {
      "label": "B",
      "content": { "title": "Hi there!", "body": "Variant B body" },
      "percentage": 40
    }
  ]
}

A successful response returns 202 Accepted:

{ "id": "job-uuid", "status": "scheduled" }

Validation errors (422 Unprocessable Entity)

The API returns 422 with a descriptive error message in the following cases:

Condition	Error message
Percentages do not sum to 100	`"variants.percentage must sum to 100"`
Variant count out of range	`"between 2 and 4 variants are required"`
`variants` used alongside `content`	`"variants and content are mutually exclusive"`
`variants` used alongside `templateId`	`"variants and templateId are mutually exclusive"`

5. Reading Variant Results

Retrieve the job details using GET /api/v1/notifications/:id. When the job was created with variants, the response includes a variantBreakdown array alongside the aggregate deliveryStats.

{
  "id": "job-uuid",
  "deliveryStats": {
    "delivered": 1000,
    "failed": 20,
    "pending": 5
  },
  "variantBreakdown": [
    {
      "label": "A",
      "variantId": "v-uuid-a",
      "delivered": 600,
      "failed": 12,
      "pending": 3
    },
    {
      "label": "B",
      "variantId": "v-uuid-b",
      "delivered": 400,
      "failed": 8,
      "pending": 2
    }
  ]
}

Field reference

Field	Description
`deliveryStats.delivered`	Total devices that successfully received the notification across both variants.
`deliveryStats.failed`	Total devices where delivery failed across both variants.
`deliveryStats.pending`	Total devices where delivery has not yet been confirmed (in-flight or queued).
`variantBreakdown[].label`	The label you assigned when creating the job (`"A"` or `"B"`).
`variantBreakdown[].variantId`	The unique ID of this variant record, useful for filtering logs or support queries.
`variantBreakdown[].delivered`	Devices that received this specific variant.
`variantBreakdown[].failed`	Devices where delivery of this variant failed.
`variantBreakdown[].pending`	Devices awaiting delivery confirmation for this variant.

Interpreting the counts

The aggregate deliveryStats counts are always equal to the sum of the corresponding fields across variantBreakdown. For example:

deliveryStats.delivered (1000) = variantA.delivered (600) + variantB.delivered (400)

To compare variant performance, look at the delivered rate for each variant: divide delivered by the total attempts (delivered + failed + pending) for that variant. A large discrepancy in failure rates between variants may indicate a content-related issue (such as a title exceeding device character limits) rather than a delivery infrastructure issue.

tip

Results are updated in near-real time as delivery confirmations arrive. If a job is still in progress, refresh the job detail page or poll GET /api/v1/notifications/:id until pending reaches 0.

6. Dashboard Walkthrough

Step 1: Enable the A/B Test toggle

Log in to the Hober Dashboard and open the Composer from the main navigation.
In the Message section, locate the A/B Test toggle and switch it on.

The Composer expands to show two content panels — one for Variant A and one for Variant B — and a percentage slider.

Step 2: Set the percentage split

Use the Split slider to set how much of your audience receives Variant A. The remaining percentage is automatically assigned to Variant B. The two percentages always sum to 100.

For example, dragging the slider to 60 gives Variant A 60% and Variant B 40%.

Step 3: Enter content for each variant

Fill in the Title and Body fields in the Variant A panel, then do the same in the Variant B panel. Both panels must be complete before you can submit.

Step 4: Submit the job

Select your target audience and channel(s) as you normally would, then click Send Notification. The job appears in Notification History with a small A/B badge indicating it is a variant job.

Step 5: Review results

Navigate to History in the left sidebar.
Locate your job and click its row to open the job detail page.
Scroll to the A/B Variant Breakdown section.

The breakdown table shows delivered, failed, and pending counts side-by-side for Variant A and Variant B. Use these counts to determine which variant performed better before promoting the winner.

Promoting the winner

If the send was created with an auto-winner experiment, the platform promotes the winner for you (or offers a one-click Send to the rest on the detail page when the result is inconclusive). For a plain variant send, promote manually:

From the job detail page, copy the winning title and body.
Open the Composer, ensure the A/B Test toggle is off, and paste the winning content.
Send as a standard notification targeting your full audience.

7. Holdout Groups

A holdout group answers a different question than an A/B test. Variants compare which message performs better; a holdout measures whether sending at all made a difference. You choose a percentage of the target audience that is deliberately not sent the message, and later compare conversion rates between the treated group and the held-out group — the gap is the send's incremental lift.

Growth plan required

Holdout groups ride the same plan feature as A/B variants. On other plans the API returns the same 403 as variant sends.

Enabling a holdout

Add holdout_percentage (1–99) to the create request. It works on both standard and variant sends; variant percentages still sum to 100 and split the treated remainder.

POST /api/v1/notifications
Authorization: Bearer <token>
Content-Type: application/json

{
  "channel_ids": ["chan-uuid"],
  "target": { "type": "all" },
  "content": { "title": "Flash sale", "body": "Ends tonight" },
  "holdout_percentage": 10
}

In the dashboard Composer, check Hold out a control group and pick a percentage (the UI ranges 1–50%; 5–20% is typical).

How holdout assignment works

Holdout bucketing is deliberately different from variant assignment in two ways:

bucket = hash(subscriber_id + job_id) % 100
bucket < holdout_percentage → held out

Per subscriber, not per device. Lift is a behavioral comparison between people; a subscriber with two devices is either fully held out or fully treated, never split.
Seeded with the job ID. Membership re-randomizes on every send, so no subscriber is permanently starved of messages the way a device permanently sticks to Variant A.

Devices without an identified subscriber are always treated — they cannot contribute to a subscriber-level comparison.

Behavior details

Held-out subscribers consume no quota and produce no delivery records; their membership is recorded before any delivery starts and survives retries.
A job whose entire (small) audience lands in the holdout completes successfully with zero sends — it is not a failure.
Holdout membership is recorded at fan-out and survives retries; assignment is deterministic per (subscriber, job).

Reading the lift

The notification detail page shows a Holdout Lift section for any job sent with a holdout: the lift delta in percentage points, and a received-vs-held-out table with group sizes, converters, conversion rates, and revenue. The same data is available from the API:

GET /api/v1/insights/lift?job_id=<job-uuid>

{
  "job_id": "job-uuid",
  "treated":  { "size": 900, "converters": 90, "conversion_rate": 0.10, "revenue_cents": 45000 },
  "holdout":  { "size": 100, "converters": 5,  "conversion_rate": 0.05, "revenue_cents": 2500 },
  "lift_percentage_points": 5,
  "window_days": 7,
  "conversion_configured": true
}

How the numbers are computed:

Each treated subscriber is anchored at their first delivered touch for the job; each held-out subscriber at the moment of assignment. A subscriber counts as converted when they have at least one conversion within the window of their anchor.
The window is the widest window across your configured conversion events (default 7 days).
Lift is the treated conversion rate minus the holdout rate, in percentage points. Negative lift is shown as-is — it means the held-out group converted more.
conversion_configured: false means you have no conversion events defined, so both sides are vacuously zero. Configure conversion events in Insights → Settings before running holdout sends, and remember that small groups make noisy lift — judge the rates alongside the group sizes.

8. Auto-Winner Experiments

An auto-winner experiment turns a variant send into a two-wave rollout: the variants go to a test slice of the audience first, the platform evaluates a metric after a window you choose, and the winning content is sent to the remainder automatically.

Creating an experiment

Add an experiment object to a variant send (it requires variants):

POST /api/v1/notifications
{
  "channel_ids": ["chan-uuid"],
  "target": { "type": "all" },
  "variants": [
    { "label": "A", "content": { "title": "Hello!", "body": "A" }, "percentage": 50 },
    { "label": "B", "content": { "title": "Hi there!", "body": "B" }, "percentage": 50 }
  ],
  "experiment": {
    "test_percentage": 20,
    "evaluation_window_seconds": 14400,
    "metric": "click_rate"
  }
}

Field	Meaning
`test_percentage`	Share of the audience in the test wave, `5`–`90`. The variants split this slice according to their percentages; the rest waits for the winner.
`evaluation_window_seconds`	How long after the (scheduled) send the evaluation runs.
`metric`	`click_rate` (default), `conversion_rate`, or `revenue_per_recipient`.

The two waves are disjoint by construction: subscribers are bucketed 0–99 per experiment, the test wave targets [0, test_percentage) and the remainder wave targets [test_percentage, 100).

Conversion and revenue metrics require an evaluation window of at least 24 hours — a shorter window would truncate attribution and reward the variant whose conversions merely arrived first. Shorter experiments must use click_rate (the API rejects the combination with a 422).

The decision rule

When the window elapses, each variant becomes an arm with its delivered count and metric events (taps for click_rate, attributed conversions for the other two; revenue_per_recipient uses refund-netted revenue). The rule:

The best metric value wins iff every arm passes the reliability gate: at least 500 delivered and 20 metric events per arm.

Anything else is inconclusive — a gate failure on any arm, an exact tie, or (for the revenue metric) arms earning in different currencies. Inconclusive means nothing auto-sends: the test wave stands, and you pick manually from the detail page. The gate is deliberately strict because a wrong winner auto-sends to the majority of your audience.

The remainder wave

The winner rollout is an ordinary notification job carrying the winning variant's content. It inherits the platform's normal delivery semantics — including quiet hours and frequency guardrails — exactly like a send you created yourself.

Experiment states

State	Meaning
`testing`	Test wave sent (or scheduled); waiting for the evaluation window.
`evaluating`	The sweep is computing the decision (transient).
`rolled_out`	A winner passed the gate; the remainder job was created.
`inconclusive`	No winner may auto-send; the stated reason is in the decision. You can still pick a winner manually.
`overridden`	You picked a winner manually; the remainder job was created.
`canceled`	You ended the experiment; the remainder is never sent.

Reading and overriding

GET /api/v1/notifications/:id includes an experiment object on experiment jobs: the state, the configuration, and — once decided — a decision with the per-arm metric values and gate verdicts. The notification detail page renders the same data as an Experiment card, with per-arm gate check marks and, for undecided or inconclusive experiments, two actions:

Send ⟨variant⟩ to the rest — POST /api/v1/experiments/:id/winner with {"variant_id": "..."}. Rolls the chosen variant out now; recorded as an override in the decision audit stamp.
Cancel — POST /api/v1/experiments/:id/cancel. Ends the experiment with no rollout.

When each action is allowed:

The winner pick works while the experiment is still running (testing, evaluating) and after an inconclusive conclusion — that second case is the designed follow-through when the gate fails: the test wave stands, nothing auto-sent, and you decide from the recorded arms. Overriding an inconclusive experiment replaces its decision stamp with the override ("mode": "override") and creates the remainder job at that moment.
Cancel works only while the experiment is still running. An inconclusive experiment needs no cancel — nothing will send unless you pick a winner.
rolled_out, overridden, and canceled are final: once a rollout happened (or was declined), no further action is accepted and the API returns 409 CONFLICT.

9. Campaign Message Variants

Campaign messages support the same machinery. On any message in a campaign, supply variants (and optionally experiment) instead of content:

POST /api/v1/campaigns
{
  "name": "Launch",
  "target": { "type": "all" },
  "messages": [
    {
      "channel_ids": ["chan-uuid"],
      "position": 0,
      "variants": [
        { "label": "A", "content": { "title": "It's here", "body": "..." }, "percentage": 50 },
        { "label": "B", "content": { "title": "Launch day", "body": "..." }, "percentage": 50 }
      ],
      "experiment": { "test_percentage": 20, "evaluation_window_seconds": 14400, "metric": "click_rate" }
    }
  ]
}

When the campaign is released, each variant message enqueues as a variant job (with its experiment, when configured) and reports per-variant results on the campaign's per-message report. variants and content/template_id are mutually exclusive per message. In the dashboard campaign builder, check A/B test this message on a message to author a two-variant split, with an optional Auto-winner configuration.

1. Overview​

2. How Variant Assignment Works​

Assignment algorithm​

Stability of the split​

3. Limitations​

4. API Example​

Create a notification with variants​

Validation errors (422 Unprocessable Entity)​

5. Reading Variant Results​

Field reference​

Interpreting the counts​

6. Dashboard Walkthrough​

Step 1: Enable the A/B Test toggle​

Step 2: Set the percentage split​

Step 3: Enter content for each variant​

Step 4: Submit the job​

Step 5: Review results​

Promoting the winner​

7. Holdout Groups​

Enabling a holdout​

How holdout assignment works​

Behavior details​

Reading the lift​

8. Auto-Winner Experiments​

Creating an experiment​

The decision rule​

The remainder wave​

Experiment states​

Reading and overriding​

9. Campaign Message Variants​