A/B Testing

An A/B experiment splits visitors between alternative htag configurations and labels their telemetry so the difference can be measured. An experiment is a name, two or more weighted variants, and the apply() each variant runs to change htag behaviour.

How It Works

Experiments are declared through runAbExperiment or runAbExperiments, normally from a RawJS snippet that runs at header start. htag picks one variant, calls its apply(), and records the assignment.

The whole selection is synchronous. The variant must be applied before htag schedules any ad, so there is no await anywhere in the path — a variant that changes lazy loading, placements, or caps has to win before the first auction is planned.

One experiment per page impression. The assignment is write-once: the first experiment that applies wins, and every later runAbExperiment call is logged and ignored. This is deliberate — overlapping treatments make the measurement unattributable.

Variant Selection

Weights are relative, not percentages: { weight: 1 } against { weight: 3 } is a 25/75 split. Selection is a plain weighted draw over the variants of the experiment.

With sticky: true, the drawn variant is remembered per visitor in the top window’s localStorage under adnz-ab-experiment-<name>. A returning visitor stays in the same bucket, so a multi-visit metric (session length, return rate) is not smeared across both arms.

The sticky value is written without a CMP consent check. That is a deliberate decision: runAbExperiment must stay synchronous, while the TCF consent lookup is asynchronous, and the two cannot be combined. What is stored is a variant key such as control — an experiment label, not a visitor identifier.

The variant is persisted only after apply() succeeded, so a variant that keeps throwing is never pinned to a visitor.

Forcing a Variant

For QA, set adnz_ab to <experimentName>:<variantKey> in the page URL or the top window’s localStorage. A matching valid variant replaces the weighted draw, and bypasses sticky bucket reads and writes. eligible() still applies, so an ineligible visitor is not enrolled. See Forcing an A/B Variant with adnz_ab for examples.

Measurement

Once an assignment exists, every collector event of that page impression carries:

• ab_experiment — the experiment name
• ab_variant — the assigned variant key
• ab_forced — 'true' when QA forced the assignment with adnz_ab
• the experiment’s optional attributes, as extra measurement tags

ab_experiment and ab_variant are reserved and always authoritative: an event producer that sets attributes with those names cannot overwrite them. ab_forced is added only for a forced assignment; htag never sets it otherwise. Because the key is simply absent from a normal session’s reserved set, an event producer that emits its own ab_forced is not overwritten there — read the marker as “this session was forced”, not as a guarantee that nothing else can set it. The attributes bag is the opposite — it is merged underneath the event’s own attributes, so a producer that emits the same key wins. Use attributes for slice labels (bucket_size, page_type), not for values an event may legitimately set itself.

Attribute Rules

• Keys — and the experiment name and variant key — are strict identifiers: ^[a-z0-9_.-]{1,64}$. This keeps grouping cardinality low.
• Attribute values are free-form strings of 1 to 256 characters.
• An invalid bag is dropped whole. The experiment still runs and is still tagged with ab_experiment / ab_variant, but the extra slice is lost and an error is logged. One bad key costs you every attribute of that experiment.

Example

A desktop-only test of the lazy-loading threshold, delivered as RawJS:

window.htag.api('1').runAbExperiments([
  {
    name: 'lazyload_threshold.v1',
    sticky: true,
    eligible: () => window.top.innerWidth >= 1024,
    attributes: { device: 'desktop' },
    variants: [
      {
        key: 'control',
        weight: 1,
      },
      {
        key: 'eager',
        weight: 1,
        apply: () => {
          window.htag.api('1').setLazyLoadingConfig((current) => current.map((config) => ({ ...config, safetyPx: 800 })));
        },
      },
    ],
  },
]);

eligible() is how a device-, geo-, or page-scoped experiment is expressed: an ineligible visitor is simply not enrolled and produces untagged events. control needs no apply() — doing nothing is the baseline.

Pitfalls

Caution

runAbExperiments treats its argument as the tag’s complete set of experiments. Sticky buckets stored for any experiment not in the list are deleted. List every experiment of the tag, including the ones that are currently ineligible or paused — dropping one from the array wipes the buckets of visitors already enrolled in it.

Version the name when you restart an experiment. Reusing image_size after changing what the variants do means every returning visitor inherits their old sticky bucket, mixing the previous run’s population into the new one. Bump it: image_size.v2.

Keep the attribute keys well-formed. Uppercase, spaces, and keys over 64 characters all invalidate the bag, and the bag is dropped as a unit — silently for the report, loudly only in the log.

A throwing apply() cancels the enrolment. htag logs the failure and records no assignment, so the page runs untreated and untagged rather than treated-but-unmeasured. Nothing is rolled back, though: whatever the variant changed before it threw stays in effect on an untagged page. Keep apply() atomic — do the work that can fail first, mutate htag last.

Related Methods

• runAbExperiment: Run a single experiment definition
• runAbExperiments: Run the first eligible experiment of a set and prune stale sticky buckets
• addRawJs: Deliver the experiment declaration with the tag