A/B Variant Sending

Premium+ plan required

A/B variant sending is available on the Premium+ plan only. You will receive a 403 PLAN_FEATURE_UNAVAILABLE error if you attempt to use variants on a Basic or Standard plan. To upgrade your plan, visit your billing settings.

A/B variant sending lets you deliver two different versions of a push notification to separate segments of your audience and compare per-variant delivery counts. You define a split — for example, 60% of devices see Variant A and 40% see Variant B — and the platform handles the assignment automatically.

This guide covers how variant assignment works, current limitations, full API examples, how to read per-variant results, and a dashboard walkthrough.

1. Overview

When you create a notification job with variants, Hober splits your target audience between two content variants according to the percentages you specify. Each device in the target receives exactly one variant. After the job runs, the notification detail page shows a per-variant breakdown of delivered, failed, and pending counts.

What A/B testing in Hober does (Sprint 11 scope):

Sends two content variants to different device segments in a single notification job
Tracks delivery counts, failures, and pending sends separately per variant
Uses a deterministic, hash-based assignment so the split is stable and reproducible

What A/B testing does not do in this release:

No automated winner promotion — if you want to follow up with only the winning variant, you must create a new non-variant notification manually
No statistical significance calculator in the dashboard (planned for a future release)
Maximum of 2 variants per job (more variants are planned for Sprint 12)

Holdout groups — withholding the send from a slice of the audience to measure incremental lift — are a separate, complementary feature. See Holdout Groups below.

2. How Variant Assignment Works

Variant assignment is hash-based and deterministic per device ID. The same device always receives the same variant for any given split percentage, regardless of when or how many times you query the assignment.

Assignment algorithm

For each device in the target audience, Hober computes:

bucket = hash(device_id) % 100

The bucket is an integer between 0 and 99 (inclusive). The device is then assigned based on the split you configured:

If bucket < variantA.percentage → the device receives Variant A
Otherwise → the device receives Variant B

For example, with a 60/40 split, devices whose hash produces a bucket of 0–59 receive Variant A, and devices with buckets 60–99 receive Variant B.

Stability of the split

Because the hash input is the device ID alone — with no job ID or timestamp component — the same 60% of devices always fall into Variant A for any job using a 60/40 split. The assignment does not shift between jobs.

This means:

Reproducible: re-running the same job delivers the same variant to the same devices.
Consistent: a device's assigned variant is predictable and auditable.
Not cross-job randomized: if you want a different 60% to see Variant A on a follow-up job, you must change the split percentages.

3. Limitations

Limitation	Detail
Exactly 2 variants required	Sprint 11 supports only 2 variants per job. Support for more variants is planned for Sprint 12.
Percentages must sum to 100	The `percentage` fields of both variants must add up to exactly `100`.
No automatic winner promotion	After reviewing results, create a new standard (non-variant) notification with the winning content to send it to your full audience.
No significance calculator	The dashboard shows raw counts only. Statistical significance analysis is not yet available.
Variants are immutable	After a job is created, its variant content and percentages cannot be changed. Create a new job to adjust the split or content.
Cannot mix with `content` or `templateId`	A job that uses `variants` must not also include a top-level `content` object or a `templateId`. These are mutually exclusive.

4. API Example

Create a notification with variants

POST /api/v1/notifications
Authorization: Bearer <token>
Content-Type: application/json

{
  "channelIds": ["chan-uuid"],
  "target": { "type": "all" },
  "variants": [
    {
      "label": "A",
      "content": { "title": "Hello!", "body": "Variant A body" },
      "percentage": 60
    },
    {
      "label": "B",
      "content": { "title": "Hi there!", "body": "Variant B body" },
      "percentage": 40
    }
  ]
}

A successful response returns 202 Accepted:

{ "id": "job-uuid", "status": "scheduled" }

Validation errors (422 Unprocessable Entity)

The API returns 422 with a descriptive error message in the following cases:

Condition	Error message
Percentages do not sum to 100	`"variants.percentage must sum to 100"`
Not exactly 2 variants	`"exactly 2 variants are required"`
`variants` used alongside `content`	`"variants and content are mutually exclusive"`
`variants` used alongside `templateId`	`"variants and templateId are mutually exclusive"`

5. Reading Variant Results

Retrieve the job details using GET /api/v1/notifications/:id. When the job was created with variants, the response includes a variantBreakdown array alongside the aggregate deliveryStats.

{
  "id": "job-uuid",
  "deliveryStats": {
    "delivered": 1000,
    "failed": 20,
    "pending": 5
  },
  "variantBreakdown": [
    {
      "label": "A",
      "variantId": "v-uuid-a",
      "delivered": 600,
      "failed": 12,
      "pending": 3
    },
    {
      "label": "B",
      "variantId": "v-uuid-b",
      "delivered": 400,
      "failed": 8,
      "pending": 2
    }
  ]
}

Field reference

Field	Description
`deliveryStats.delivered`	Total devices that successfully received the notification across both variants.
`deliveryStats.failed`	Total devices where delivery failed across both variants.
`deliveryStats.pending`	Total devices where delivery has not yet been confirmed (in-flight or queued).
`variantBreakdown[].label`	The label you assigned when creating the job (`"A"` or `"B"`).
`variantBreakdown[].variantId`	The unique ID of this variant record, useful for filtering logs or support queries.
`variantBreakdown[].delivered`	Devices that received this specific variant.
`variantBreakdown[].failed`	Devices where delivery of this variant failed.
`variantBreakdown[].pending`	Devices awaiting delivery confirmation for this variant.

Interpreting the counts

The aggregate deliveryStats counts are always equal to the sum of the corresponding fields across variantBreakdown. For example:

deliveryStats.delivered (1000) = variantA.delivered (600) + variantB.delivered (400)

To compare variant performance, look at the delivered rate for each variant: divide delivered by the total attempts (delivered + failed + pending) for that variant. A large discrepancy in failure rates between variants may indicate a content-related issue (such as a title exceeding device character limits) rather than a delivery infrastructure issue.

tip

Results are updated in near-real time as delivery confirmations arrive. If a job is still in progress, refresh the job detail page or poll GET /api/v1/notifications/:id until pending reaches 0.

6. Dashboard Walkthrough

Step 1: Enable the A/B Test toggle

Log in to the Hober Dashboard and open the Composer from the main navigation.
In the Message section, locate the A/B Test toggle and switch it on.

The Composer expands to show two content panels — one for Variant A and one for Variant B — and a percentage slider.

Step 2: Set the percentage split

Use the Split slider to set how much of your audience receives Variant A. The remaining percentage is automatically assigned to Variant B. The two percentages always sum to 100.

For example, dragging the slider to 60 gives Variant A 60% and Variant B 40%.

Step 3: Enter content for each variant

Fill in the Title and Body fields in the Variant A panel, then do the same in the Variant B panel. Both panels must be complete before you can submit.

Step 4: Submit the job

Select your target audience and channel(s) as you normally would, then click Send Notification. The job appears in Notification History with a small A/B badge indicating it is a variant job.

Step 5: Review results

Navigate to History in the left sidebar.
Locate your job and click its row to open the job detail page.
Scroll to the A/B Variant Breakdown section.

The breakdown table shows delivered, failed, and pending counts side-by-side for Variant A and Variant B. Use these counts to determine which variant performed better before promoting the winner.

Promoting the winner

To send the winning variant to your full audience:

From the job detail page, click Use as template on the winning variant panel (or copy the title and body manually).
Open the Composer, ensure the A/B Test toggle is off, and paste the winning content.
Send as a standard notification targeting your full audience.

7. Holdout Groups

A holdout group answers a different question than an A/B test. Variants compare which message performs better; a holdout measures whether sending at all made a difference. You choose a percentage of the target audience that is deliberately not sent the message, and later compare conversion rates between the treated group and the held-out group — the gap is the send's incremental lift.

Premium+ plan required

Holdout groups ride the same plan feature as A/B variants. On other plans the API returns the same 403 as variant sends.

Enabling a holdout

Add holdout_percentage (1–99) to the create request. It works on both standard and variant sends; variant percentages still sum to 100 and split the treated remainder.

POST /api/v1/notifications
Authorization: Bearer <token>
Content-Type: application/json

{
  "channel_ids": ["chan-uuid"],
  "target": { "type": "all" },
  "content": { "title": "Flash sale", "body": "Ends tonight" },
  "holdout_percentage": 10
}

In the dashboard Composer, check Hold out a control group and pick a percentage (the UI ranges 1–50%; 5–20% is typical).

How holdout assignment works

Holdout bucketing is deliberately different from variant assignment in two ways:

bucket = hash(subscriber_id + job_id) % 100
bucket < holdout_percentage → held out

Per subscriber, not per device. Lift is a behavioral comparison between people; a subscriber with two devices is either fully held out or fully treated, never split.
Seeded with the job ID. Membership re-randomizes on every send, so no subscriber is permanently starved of messages the way a device permanently sticks to Variant A.

Devices without an identified subscriber are always treated — they cannot contribute to a subscriber-level comparison.

Behavior details

Held-out subscribers consume no quota and produce no delivery records; their membership is recorded before any delivery starts and survives retries.
A job whose entire (small) audience lands in the holdout completes successfully with zero sends — it is not a failure.
Holdout membership is recorded at fan-out and survives retries; assignment is deterministic per (subscriber, job).

Reading the lift

The notification detail page shows a Holdout Lift section for any job sent with a holdout: the lift delta in percentage points, and a received-vs-held-out table with group sizes, converters, conversion rates, and revenue. The same data is available from the API:

GET /api/v1/insights/lift?job_id=<job-uuid>

{
  "job_id": "job-uuid",
  "treated":  { "size": 900, "converters": 90, "conversion_rate": 0.10, "revenue_cents": 45000 },
  "holdout":  { "size": 100, "converters": 5,  "conversion_rate": 0.05, "revenue_cents": 2500 },
  "lift_percentage_points": 5,
  "window_days": 7,
  "conversion_configured": true
}

How the numbers are computed:

Each treated subscriber is anchored at their first delivered touch for the job; each held-out subscriber at the moment of assignment. A subscriber counts as converted when they have at least one conversion within the window of their anchor.
The window is the widest window across your configured conversion events (default 7 days).
Lift is the treated conversion rate minus the holdout rate, in percentage points. Negative lift is shown as-is — it means the held-out group converted more.
conversion_configured: false means you have no conversion events defined, so both sides are vacuously zero. Configure conversion events in Insights → Settings before running holdout sends, and remember that small groups make noisy lift — judge the rates alongside the group sizes.

1. Overview​

2. How Variant Assignment Works​

Assignment algorithm​

Stability of the split​

3. Limitations​

4. API Example​

Create a notification with variants​

Validation errors (422 Unprocessable Entity)​

5. Reading Variant Results​

Field reference​

Interpreting the counts​

6. Dashboard Walkthrough​

Step 1: Enable the A/B Test toggle​

Step 2: Set the percentage split​

Step 3: Enter content for each variant​

Step 4: Submit the job​

Step 5: Review results​

Promoting the winner​

7. Holdout Groups​

Enabling a holdout​

How holdout assignment works​

Behavior details​

Reading the lift​