Group A/B testing takes popup experimentation to the next level by letting you test an entire group of popups against another group. Rather than comparing a single popup to a single popup, group testing lets you evaluate full acquisition strategies head-to-head so you can get clearer metrics on what's actually driving lift.
NOTE: Group A/B testing replaces the previous Popup A/B Testing flow. You can still run a traditional one-on-one test — just assign a single popup to each visitor group's popup set, and you'll have the same head-to-head comparison as before.
Group testing simply adds the option to test multiple popups per side so you can also answer bigger strategic questions, such as:
Does a cashback incentive strategy outperform a discount strategy across all touchpoints?
Is a streamlined set of two broad popups more effective than ten hyper-targeted ones?
In this article, we'll discuss how to set up a group A/B test and how to read your results.
What Is Group A/B Testing?
In a group A/B test, you assign a popup set to each visitor group — for example, multiple popups to Group A (the control) and multiple popups to Group B. Each visitor group's popup set can include a single popup or multiple popups, so you can run anything from a classic one-popup-vs-one-popup test to a full strategy comparison. You must have at least two visitor groups and can add up to four, so you can compare several strategies at once (for example, 2 popups vs. 5, or 10 vs. 2).
When a visitor lands on your site, Postscript checks each page they visit for an active group test. If a visitor lands on any included page of any popup in the test, they're randomly assigned to a visitor group based on your distribution settings, and will only ever see popups from that group's set for the duration of the test. Within their assigned group, popups still fire according to each popup's own targeting and triggers (time delay, exit intent, audience, etc.). At the end of the test, you can see which visitor group's strategy drove more opt-ins.
Here are a couple of examples:
Incentive strategy test (e.g., discount vs. cashback):
Group A (control): Evergreen popup with discount + exit intent popup with discount
Group B (test): Evergreen popup with cashback + exit intent popup with cashback
Multi-popup UTM targeting test:
Group A (control): 10 popups, one per collection page being targeted by different ads
Group B (test): 2 popups — one for all men's products and one for all women's products
Set Up a Group A/B Test
Before you begin, make sure that all popups you want to include in the test are already created and activated. You'll also want to configure each popup's targeting (pages, audience, etc.) before starting the test, as include/exclude pages can no longer be overridden at test creation time.
NOTE: You can no longer override a popup's include/exclude pages at A/B test creation time. Postscript now relies on each popup's own targeting criteria to determine who sees what, so set your page targeting on each popup individually before starting the test.
Select Acquisition in the side menu of your Postscript dashboard. Select Popups.
Select Create in the upper right corner, then select A/B Test.
Name the A/B test. Choose a name that will allow you to easily identify and track this test in the future.
Under Distribution and Popup Selection, add the popups you'd like to assign to each visitor group's Popup set. A popup set can include a single popup or multiple popups. Use + Add Visitor Group to add more groups — you must have at least two and can have up to four.
Set the Distribution percentage for each visitor group — this controls what share of visitors will see each group's popups, and the total must equal 100%. Use the Auto Balance button to split traffic evenly across groups.
Under Targeting, note that targeting is defined by each popup within the test — it can no longer be overridden here. Visitors who land on any included page of any popup in the test will be randomly added to one of the visitor groups based on your distribution. Confirm each popup's page targeting is what you expect before starting.
Under Test Duration, choose when the test should start (Immediately or Schedule the test for a later time) and when it should end (Run until the test is stopped manually or Pick a date to end the test and promote the winner afterward). Declaring a winner will keep the most successful group's popups active and switch the others to inactive.
Once you've configured your visitor groups and settings, select Start Test in the upper right corner.
Read Your Group Test Results
After your test has run, head to the test results page to see how each visitor group performed. Results now combine mobile and desktop data into a single unified view, and % lift over the control is displayed directly in the app — no manual math required. A more detailed breakdown of each visitor group, including the individual popups within it, is available below the summary.
Postscript determines the winning visitor group based on which has the best cumulative opt-in conversion rate relative to its traffic distribution. The visitor group with the higher conversion rate wins.
Get Support
Have questions? Please feel free to reach out to our Support team at [email protected] or via live chat.
Need ongoing channel strategy guidance? Please fill out this form and we'll connect you to one of our certified partners.