Introduction to A/B Testing

A/B testing is a powerful tool to help write effective messages with high customer engagement.

The idea is simple: 

  1. Write two (or more) different messages.
  2. Send each message to different small samples of customers. 
  3. Compare results. Whichever message has the highest EPR and CTR is the most effective.
  4. Send this message to the rest of your customers. 

A/B testing has long been used in marketing as a way to make writing much more scientific and much less guesswork. Now, Postscript gives you the power to use A/B testing. This guide explains how.


What criteria can be A/B Tested for Campaigns?

  1. Text -- will different writing produce higher response rates?
  2. GIFs/Images -- will different GIFs/images produce higher response rates?
  3. Coupons -- will higher discounts/sales produce higher response rates?

__________________________________________________________________

A/B Testing Campaigns Walk through

Note: This may seem like a lot of steps, but it should take less than 10 minutes (outside of writing the messages and waiting 24 hours for the data)

Here's the plan...

  • Creating the Segments: Segment your list into 3 different segments using the A/B Test Group feature. Create 2 small sample groups and a main group.
  • Sending the Tests: Send different campaigns to each sample group.
  • Analyzing the Better Response: Wait 24 hours and analyze which campaign was more effective
  • Send the Better Response to the Main Group: Send that message to the main group.

Step-By-Step Guide

Creating the Segments

  1. Click "Segments" on the left
  2. Click "Create Segment" on the top right
  3. Click on the "Number of Orders" box. A dropdown will pop up. Click "A/B Test Group" at the top.

4. Click the "2 Groups" Box and change that to "5 Groups"
5. Under "Segment Name", name this segment "[name of campaign] Sample 1"
6. Click "Save Segment" on the bottom right

congrats, you've now created your first sample. let's make the second one.

7.  Repeat steps 1-4
8. Click the "Group 1" box and change that to "Group 2"
9. Under "Segment Name", name this segment "[name of campaign] Sample 2". Then, click save segment

now we have our samples, let's make the big group. (shortcut: copy the picture below step 16)

10. Repeat steps 1-4
11. Click the "+ OR" box twice.
12. Change both "Number of Orders" boxes to "A/B Test Group" (there should now be 3 boxes that say A/B Test Group)
13. Change the top boxes to "Group 3" out of "5 Groups"
14. Change the middle boxes to "Group 4" out of "5 Groups"
15. Change the bottom boxes "Group 5" out of "5 Groups"
16. Under "Segment Name", name this segment "[name of campaign] Main Group". Then, click save segment

you've made your groups. now lets send the tests

Sending the Tests

  1. Click "Campaigns" on the left
  2. Click "Create Campaign" on the top right
  3. Below Segment, change the box to your FIRST sample group i.e. "[name of campaign] sample 1"
  4. Below Campaign Name, name the campaign "test campaign 1"
  5. Write your message, include your links/discount codes, and optionally add a GIF/Image (take your time with this! be creative! this is what your customers will see!)
  6. Click "Save & Continue to Preview" on the bottom right


7. Click "Send Campaign Now" on the bottom right


you just sent your first test to sample group 1. Let's repeat for sample group 2.

  1. Click "Campaigns" on the left
  2. Click "Create Campaign" on the top right
  3. Below Campaign Name, name the campaign "test campaign 2"
  4. Below Segment, change the box to your SECOND sample group i.e. "[name of campaign] sample 2"
  5. Write a second message.
  6. Click "Save & Continue to Preview" on the bottom right
  7. Click "Send Campaign Now" on the bottom right

congrats. you've sent your test campaigns. lets wait a bit to get some results, then see which one was better?

Analyzing the Better Response

  1. Wait 24-72 hours.
  2. Click "Campaigns" on the left
  3. Under sent campaigns, analyze the data to see which campaign was more successful. 

As a rule of thumb, we want:

  • Lower Unsub.
  • Higher CTR
  • Higher Conv.
  • Higher EPR

For example, in the picture below, we would want to pick the campaign on the top because it has a lower unsub, higher CTR, higher conv, and higher EPR.

Once you've selected the better campaign, we want to send it out to everyone else!

Send the Better Response to the Main Group

  1. Click "Campaigns" on the left
  2. Below Sent Campaigns click the name of the better test campaign (it will be in light blue writing)
  3. Click "Clone" on the top right
  4. Rename this campaign to "[name of campaign] Main"
  5. Below Segment change the box to "[name of campaign] Main Group"
  6. Click "Save & Continue to Preview" on the bottom right
  7. Click "Send Campaign Now" on the bottom right

Voila! You just successfully sent an A/B Tested Campaign to your customers. Now you know that the message you sent was tested scientifically for better responses.

If you have any questions please let us know in the chat! We're here to help.


Did this answer your question?