Ultimate Cold Email A/B Testing Guide

Andy Cabasso

January 7th

Outbound and inbound sales are like a pair of squabbling siblings.

Inbound is the younger one. It gets all the praise. And it never gets the blame when something goes wrong.

But just because inbound is the golden child, that doesn’t mean you ignore outbound. You love them both equally, right?

And there’s definitely a lot to love about outbound sales, because despite what a lot of marketers claim, cold email outreach and cold calling are still super effective.

That’s why the average salesperson sends more than 36 emails a day.

Still, there are no guarantees.

If your sales reps are spending a big chunk of time writing email content, they need to generate results. That means understanding what works (and what doesn’t).

A/B testing is the answer. In this guide, you’ll learn everything you could ever need for running effective cold email A/B tests to turbocharge your cold email campaigns.

What is A/B testing?

Also known as “split testing” and “bucket testing”, A/B testing involves comparing two (or more) variants of a “thing” against one another to discover which works best.

That “thing” could be a website form, an app screen, a cold email subject line, or something else entirely.

The two variants are simultaneously shown to different user segments, whose behavior and engagement are measured to understand whether the new variant has a positive, negative, or neutral impact.

Ideally, your variants will only differ in one small, specific way. You wouldn’t change an email’s subject line and preview text then test it against the original email, because it’d be impossible to understand the impact of each individual alteration.

For instance, Dutch mobile service provider Ben ran an A/B test on its product pages, changing the positioning of its color selection tool from directly beneath the product image…

Original (“control”) variant

…to alongside the product as a separate menu.

Test variant

That might seem like a pretty inconsequential change, but the new variant delivered a 17.63% increase in conversions.

Benefits of Cold Email A/B Testing

No one conducts split-test email campaigns for the hell of it; they do it to optimize performance. From a cold outreach perspective, they’re looking to improve one of the following metrics:

1. Open Rate

Testing your subject lines and preview text will help you understand what “good” variants look like, enabling you to optimize campaign performance.

According to Mailchimp, the average open rate across all industries stands at 22.71%, while Campaign Monitor cites a slightly lower figure of 18%.

2. Click Rate

Is your audience more likely to click on text-based CTAs or banner images?

Do readers prefer certain words or phrases?

A/B test these elements to improve your email click rate.

When it comes to benchmarks, Campaign Monitor and Mailchimp give near-identical figures of 2.6% and 2.9% respectively.

3. Response Rate

Your goal might not be to drive clicks to your website; you might be looking for an immediate response from your prospect instead.

Another cold email statistic suggests that an average reply rate benchmarks are hard to come by, but estimates range from 10% to 30%.

As a rule of thumb, if you see a higher-than-average open rate, your reply rate should be above-average too.

Which Elements of Cold Email to Split Test

You can play around with multiple elements when split testing cold email campaigns. Here are some of the most common variants:

1. Subject Line

As the saying goes, you only get one chance to make a first impression.

Your subject line (and, to a lesser extent, your preview text) is your big opportunity to stop recipients in their tracks while scrolling through their busy inbox. In fact, 34% of people open emails based on the subject line.

Clearly, split testing subject lines make a lot of sense.

2. Opening Line

The average person receives over 330 emails a day, so your intro copy needs to pique their interest immediately — otherwise, they’ll bounce straight back to their inbox. Every cold outreach automation tool has the option to insert personalized opening lines.

Personalized opening lines are especially impactful, but don’t just say “Hi {first.name}”.

Go deeper by referencing their industry, job role, or a key pain point they’re experiencing.

3. Value Proposition

Your value proposition is the “secret sauce” that separates you from your rivals.

While your product or service doesn’t change, the value you provide — and the way you communicate it — might vary depending on your audience.

Here at Postaga, we target sales teams by discussing our cold outreach functionality, while when we’re speaking to marketers, we’ll talk about things like link-building and blog post promotion.

4. Personalization

The “P” word again. Personalization is super important to cold email prospecting.

Emails with “advanced” personalization (i.e. those that go further than referencing the user or company name) see more than double the response rate of emails without advanced personalization.

Try split testing the way you add personalization. Does it work best in the subject line? In the body copy? Or even in images or videos?

5. Email Body

There are any number of ways to A/B test your email body, from your tone of voice to the length of your copy and the way it’s formatted. For instance:

Does your audience prefer a bit of humor or something more straight-laced?
Do they want to read multiple paragraphs or just a couple of short sentences?
Do they like long sentences or short, bullet-pointed lists?

Split testing will give you the answers!

6. Call to Action

If your Call To Action isn’t ultra-clickable, you won’t see results from cold email outreach: it’s as simple as that.

Test out different action words and tones of voice; test CTA buttons vs. text links; test the location of your calls to action (at the end of your email? Within the body copy?).

How to A/B Test Your Cold Emails

Higher open rates, increased click-throughs, more responses… running A/B tests in email sounds pretty good, right? Now, let’s dive into the step-by-step process for split testing.

Step 1: Choose One Element

Make your A/B tests as specific and actionable as possible by focusing on a single element for each test.

For instance, you might send identical emails with different subject lines, or test two different CTAs to see which delivers more click-throughs.

Step 2: Create Different Versions Elements

You’ve chosen which element to test. Now, create two different variants of that element. Simple!

Step 3: Set Campaigns for Each Version

Next, it’s time to build your split test email campaigns.

Treat each variant as a unique campaign, because you don’t want to send both emails to your entire email list.

Your best bet is to do lead segmentation (or one specific segmented list at a time) in half, randomly — that way you’re not introducing another variable to your tests.

Step 4: Test Follow-Ups

To be clear, this isn’t about testing whether you should send follow-up emails or not at all (hint: you definitely do). Instead, test things like:

The number of automatic follow-up emails you send
The delay between sending each follow-up email
The time of day at which follow-up emails are sent

Step 5: Check the Campaign Stats

In email split testing as in life, there are winners and losers. Crunch the numbers — open rate, click rate, and reply rate — to understand which variant worked best.

Do you have enough results to draw statistical significance from the data? Were the sample sizes large enough?

If not, repeat the test in your next cold email campaign.

How to A/B Test Elements to Increase Open Rate

Remember, benchmarks for email marketing campaigns reveal an average response rate of around 20%.

Whether your current email open rate is above, below, or in line with that benchmark, it’s definitely in your interest to improve it through A/B testing. Here’s how:

Audit Your Email Account

If your email account is in poor shape, all the split testing in the world won’t improve your open rates.

That’s why the first step is to assess your email account health by:

Checking your spam score using a free tool like Sender Score
Checking your email deliverability rate for free using MxToolbox
Setting up SPF, DKIM, and DMARC records, which can help your deliverability and increase your chances of reaching inboxes rather than spam folders
Warming up a new email address or domain to avoid raising suspicion from other email servers

Personalize the Subject Line

According to Campaign Monitor, personalized email subject lines are 26% more likely to be opened.

But don’t trust third-party statistics; see for yourself. Does your audience respond better to personalized or non-personalized subject lines?

Ask a Question in the Subject Line

Questions can be a powerful tool to engage recipients and compel them to click.

Research from Yesware discovered that asking questions in subject lines improved open rates by 8.7 percentage points:

Again, just because it worked for Yesware, that doesn’t mean it’ll work for you.

Test a question-based subject line alongside a “standard”, question-free subject line and see which your audience prefers.

Figure Out the Best Day & Time

Plenty of research has been performed into the best day and time to send emails.

For instance, Sendinblue claims the best times to send an email are:

From 9:00 am – 10:00 am
Around 2:00 pm

While Campaign Monitor says emails sent on Fridays see higher open rates than those sent on other days.

But maybe your audience can’t resist a Thirsty Thursday, meaning Friday mornings aren’t a great time to reach them? Split testing will help you find the perfect window for maximizing open rates.

Additionally, bear in mind these two best practices:

It’s best not to send emails at weekends or during holidays
Be sure to schedule email marketing campaigns around your audience’s time zone

Alter Your Subject Line Format

While you’ve only got about 40 characters to play with, there are a couple different ways to format your email subject lines. Try testing:

Words in all-caps: This can add impact (but don’t do it all the time; no one likes being shouted at repeatedly).
All lowercase subject lines: Looks cool and casual. But maybe your audience prefers to keep things professional?
Emojis in subject lines: Campaign Monitor saysbrands that use emojis enjoy a 56% upturn in unique open rates. However, emojis definitely aren’t for everyone! If you’re going to trial them, be sure to maintain a casual, humorous tone in your email body copy.

Use Power Words in the Subject Line

Power words are highly persuasive words designed to trigger an emotional or psychological response from your audience.

There are literally thousands of power words to choose from, but common examples include:

Bargain
Easy
Free
Genius
Kickstart
Quick
Secrets

As ever, different tactics work for different audiences. Experiment to find power words and phrases that your email recipients simply can’t resist.

Add Bespoke Preview Text

Your sender name and subject line aren’t the only elements that display within the main inbox; pretty much every email client also shows up to 278 characters of preview text.

Here’s how preview text displays in a standard Gmail desktop inbox and on mobile.

Think of preview text as a short summary of your email body, working alongside the subject line to compel recipients to click.

Test different approaches to your preview text, such as personalizing the copy or adding a question.

How to A/B Test Elements for Higher Click Rate

Getting plenty of opens, but not a lot of people clicking through to your website? Here are some elements you can test to boost your email click rate.

Test Different Link Volumes

Where links are concerned, it’s rarely a case of “the more, the merrier”.

Too many links in your emails likely means you’re asking recipients to take too many different actions. One action per email is plenty.

However, you might see success from including a link at the top of the email and another CTA at the end.

Switch Up Your Link Placements

If you’re adding two links to the same location, it’s easy: add one near the start and the other at the end.

But if you’re only going with one link, it needs to be in the most impactful location possible.

That could be in the body copy, in the intro, or in a “P.S.” at the end of the message.

Use Different Anchor Text Links

First off, don’t paste the full URL into your email. You might think you’re being 100% transparent, but URLs look hideous and they’re difficult to parse.

Instead, write descriptive and engaging anchor text that explains what’s on the other side of the link and compels readers to click, just like this example from Authority Hacker:

Make Sure Your Links Are Working!

This point doesn’t require any further explanation — just promise me you’ll check your link(s) before hitting “send” (I’ve definitely learned this lesson the hard way).

How to A/B Test Cold Email to Get More Interested Responses

Not interested in click-throughs?

Specifically looking for an actual response from your email recipients?

Test these elements to run more effective email marketing campaigns with higher reply rates.

Ask When to Follow Up

If the goal of your email campaign is to arrange a meeting or call, make it easy for your audience to respond by asking what works best for them.

There are various ways you can do this. Try:

Suggesting a specific date and time, and asking if it suits them
Sending your preferred time and asking for theirs
Asking for their favored time
Linking to a tool like Calendly

Reiterate Your Unique Value Proposition

Generally speaking, there’s no harm in overstating your value, making it totally clear what benefit your product or service brings and why it’s the ideal fit for your prospects.

However, if you’re five emails deep into a sequence, making the same point over and over again might wear a little thin.

Play around with different ways of expressing your value proposition and whether to include or exclude it from certain emails in your sequence.

Sharing Valuable Advice or Information

Follow-up emails aren’t simply an opportunity for you to chase a response — they need to add their own value.

Maybe it’s a useful blog you’ve written or a piece of research you’ve come across.

It could be a free tool or an interesting statistic.

Whatever the case, test sharing that information freely vs. directly asking recipients if it’s something they’d be interested to learn.

With a proper follow-up email strategy, you nail down getting responses.

Don’t Forget To Add Questions

I’ve already mentioned how questions can be super effective in email subject lines, but they can work just as well within the body copy.

As ever, there are various tests you can run here, such as:

Asking a question vs. no question
Asking a closed question vs. an open-ended question
Asking a question in the main body text vs. in a “P.S.” at the bottom

Testing Different Email Templates

While email A/B testing is typically about small variations, sometimes a broader approach is required.

If nothing else is working, your cold email templates might be the problem. In which case, it’s time to go back to the drawing board.

Refresh your campaign by:

Segmenting your prospects based on pain points
Splitting the list in half at random
Writing two different email templates with identical subject lines
Seeing which version generates the highest response rate

7 Best Practices for Running a Cold Email A/B Test

That’s pretty much everything you need to know to get your first split test off the ground.

But before you dash off to test your first campaign, make sure you’re following these best practices:

Always segment your email list before running tests. Dividing it in half, at random, is just fine.
Build a highly targeted audience. Cold email works best when it’s a little “warmer”. That means researching and pre-qualifying your prospects before adding them to a campaign.
Don’t be afraid to experiment. Just because you’ve always done something a certain way, there could still be a better way to do it. Nothing is off the table in A/B testing!
Change one element at a time. That way, you can clearly measure the impact of your test.
Prioritize elements to test. There are likely dozens of tests you could run. Start by identifying as many as possible, then prioritize the elements that are likely to have the biggest impact (hint: subject lines are a good starting point).
Learn from the previous tests. If an earlier test revealed that your audience hates emojis, steer clear of them in future tests!
Test your signature. Try editing your email signature by adding links, images, or short testimonials, then measure the impact on click-throughs and replies.

Wrapping It All Up

There are rarely any silver bullets in cold email outreach.

I highly doubt you’re going to run one A/B test and immediately double your open, click-through, or response rates.

Rather, it’s a game of incremental gains.

Each test might only net you a 1% increase in whichever metric you’re targeting, but those tiny improvements make a big difference over time.