A/B testing is available for customers who have purchased Nutshell Campaigns. All Nutshell users can send up to 150 emails per month with Campaigns for free before adding it to their plan.
Ever wondered about the most effective way to connect with your audience through email? With A/B testing, you can send two different email versions to a small subset of your audience segment.
The ultimate goal? Identify the version that resonates best with your audience based on your selected key metric like clicks or open rate. Once the winning version is determined, Nutshell will automatically send it to the remaining audience. This approach is particularly advantageous as the rest of the audience is expected to respond positively to the winning version, as evidenced by the test results.
Let's dive into the details of how to set A/B tests up.
Get started
Access A/B test
You can set up A/B tests for Broadcasts and Newsletters in Nutshell. Simply go to the Marketing menu, and you'll find the A/B test tab within the Broadcasts and Newsletters options.
Create an A/B test
You have the option to create an A/B test from a new broadcast/newsletter or you can enable an A/B test from an existing standard broadcast. To create an A/B test from scratch, simply select “A/B test” as the broadcast type in the dialogue below.
If you’re setting up an A/B test from an existing standard broadcast/newsletter, simply click the ‘Enable A/B test’ button on the A/B test tab:
Select your A/B test versions
You have the option to choose between one or all three elements to experiment with— subject lines, preview text, and content. Simply click the respective checkbox for your test as seen in the image below.
Content, subject line, and preview text options
Create content versions
Click on the Editor tab to create two versions of your content. From the dropdown, switch between Version A and B to make the respective content changes.
If you'd like to make the same changes to Version B or vice versa, simply click the Transfer content between versions text button.
When clicked, the dialogue below will appear which will allow you to click the version you'd like to transfer the content and design from. This way, you can effortlessly replicate the design and content from one version to another, saving you from repetitive design work. For instance, if you want the design to stay the same but only tweak a line or two of text, the Transfer content between versions text button will be incredibly useful in this situation.
Create subject line and preview text versions
On the Details tab, you can add the two subject lines and preview text. If you'd only like to text the subject line, simply toggle the preview text off or vice versa.
Additionally, on the Details tab specify the ‘from’ email address and the audience to which you'd like to send the test.
Configure the A/B test
Head to the A/B test tab to configure the test. This is the control center where you will set the criteria for your test. This includes:
Set the winning version metric
You can decide the winning criteria from the test—either the highest open rate or the highest click rate.
Set test distribution percentage
Choose the audience size that aligns with your testing goals. Set the subset of the audience for the test from the available distribution percentage options.
Note: 20 or more recipients are required for A/B tests to be sent successfully. For more impactful results, it's recommended to send A/B tests to an audience list of 1000+ recipients. However, you can still send it to a list with fewer than 1000 people.
Set test duration
Choose when the winner will be declared after sending the initial batch of emails for comparison. Essentially, Nutshell sends the two versions to a small subset of your audience. After the test concludes based on the test duration, the winning version will be sent to the rest of the audience. Options include:
- 2 hours
- 4 hours
- 8 hours
- 24 hours
Set a fallback version
In the event of a tie, you can choose the version of the email that will be sent to the rest of the audience.
Preview comparison
After setting up, compare the versions in the quick preview on the left side of the A/B test tab. You’ll see the subject line, preview text, and the content in a card below each other.
Send an internal test
Once you're happy with your A/B test and ready for a final preview, you can send a test to yourself and/or team members by clicking the "Send test emails" button. Both versions A and B will be sent as separate emails for a thorough review.
Share email URL
Additionally, you can also easily share email versions with your team by clicking the Share URL button and copying the URLs for them to view the emails online.
Send the A/B test
When you are officially prepared to start the A/B test, click the "Prepare to send" button.
You’ll be able to confirm that the following details are correctly added:
From address
To audience list
Subject line
Unsubscribe link
Mailing address
After confirming, proceed to review the test. Clicking the Review button will allow you to take one last look at a quick preview of the two versions, the audience distribution test split, the broadcast name, the from address, and the audience details.
After the final review, click the Send A/B test button to officially send the configured A/B test versions to your selected audience.
View engagement results
As the test runs, keep an eye on the Engagement tab. It'll show you which version is currently ahead. When the test finishes and a winner is decided, the winning version will be clearly labeled.
You can also check detailed results for both versions, including message delivery, open rate, click rate, unsubscribe rate, and bounce rate. This helps you understand how each version performed.
FAQs
Why should I use A/B testing in Nutshell for my email campaigns?
A/B testing in Nutshell allows you to optimize your email content based on actual audience responses. By comparing different versions and identifying the one that resonates best, you can improve engagement, click rates, and overall campaign effectiveness.
How do I choose which elements to A/B test?
You can choose to test the subject line, preview text, and/or the content—simply ensure the toggle is turned on.
Can I track the performance of each A/B variation?
Yes, you’ll be able to see the test results on the Engagement tab.
How is the winning version determined?
Nutshell automatically declares the winning version based on the metric you selected when configuring the test.