Introduction to A/B Testing

A/B testing is a powerful tool to help write effective messages with high customer engagement.

The idea is simple: 

  1. Write two (or more) different messages.
  2. Send each message to different small samples of customers. 
  3. Compare results. Whichever message has the highest EPR and CTR is the most effective.
  4. Send this message to the rest of your customers. 

A/B testing has long been used in marketing as a way to make writing much more scientific and much less guesswork. Now, Postscript gives you the power to use A/B testing. This guide explains how.

What criteria can be A/B Tested for Automations?

  1. Text -- will different writing produce higher response rates?
  2. GIFs/Images -- will different GIFs/images produce higher response rates?
  3. Coupons -- will higher discounts/sales produce higher response rates?
  4. Timing -- how often should these automations be going out?
  5. Type of Message -- MMS or SMS work better for your customers?
  6. Automations have a ton of filters available that you can test between. This is only a sampling of your options!

__________________________________________________________________

A/B Testing Automations Walk through

Here's the plan...

  • Setting up the Automations: Edit your messages and filter your A/B groups
  • Sending the Tests: Enable those automations to start sending!
  • Analyzing the Better Results: Let the automations run for several days/weeks depending on the amount of subscribers you have and analyze which automation was more effective
  • Choose the Automation with the Best Results: That's the automation that you'll want to keep enabled and have running for your customers. 

Step-By-Step Guide

Setting up the Automations

  1. Click "Automations" on the left
  2. Click "Create Automation" in the top right corner (You can name it A/B Test 1)
  3. Use the dropdown for User Filter to choose "Yes"
  4. Select "A/B Test Group" 

5. Decide how many A/B tests you want to run at once. You must create an          automation for each group. In this example, we'll be running a 2-group A/B test.

6. Set up the Automation Trigger and Cancellation Trigger to your liking for the specific automation that you're running. Then click "Save & Continue to Messages"
7. Edit the body of your text as you need - keeping in mind the criteria that you're testing (Text, GIF/Image, coupon, etc)
8. Save Automation

You would then go back to your Automation page as mentioned in step #1 and repeat the process. 

When you reach step #2, choose Group 2 out of 2 Groups. 

When you are editing the body of the text for this test, make the criteria that you're testing different from the first that you set up. 

Sending the Tests

From your Automations tab on the left, you'll see all of your automations. To make them live, simply click Enable

Congrats. You've sent your test automations. Lets wait a bit to get some results, then see which one was better.

Analyzing the Better Results

  1. Wait until you've sent 500-5000 texts (depending on your customer size)
  2. Click "Automations" on the left
  3. Next to each automation, analyze the data to see which automation was more successful. 

As a rule of thumb, we want:

  • Lower Unsub.
  • Higher CTR
  • Higher Conv.
  • Higher EPR

For example, in the picture below, we would want to pick the automation on the top because it has a lower unsub, higher CTR, higher conv, and higher EPR.

*This example was split out after the first automation had been running for awhile. 

Once you've selected the better automation, we want to send it out to everyone else!

Choose the Automation with the Best Results

  1. Click "Automations" on the left
  2. Click on the better-performing automation & choose Edit in the top right corner
  3. Choose "Edit Automation Properties" on the bottom right.
  4. Rename that automation (Since it's no longer a A/B Test)
  5. Remove the A/B Test Group User Filter & allow it to go out to the filters that you prefer. 
  6. Save that automation & be sure to Disable the other A/B Test automation!

Voila! You just successfully sent an A/B Tested Automation to your customers. Now you know that the message you sent was tested scientifically for better responses.

If you have any questions please let us know in the chat! We're here to help.

Did this answer your question?