If you’re sending cold emails and just guessing what works, you’re probably wasting your time—and your leads. The only way to really know what gets replies is to test it. That’s where A/B testing comes in. This guide is for people who want real answers, not marketing fluff, and want to use Scaledmail to do it.
Here’s how to set up A/B testing for cold emails in Scaledmail step by step, minus the hype. We'll cover what actually matters, what to skip, and a few common mistakes you can avoid.
Why Bother With A/B Testing Cold Emails?
Let’s be blunt: Most cold emails get ignored. If you aren’t testing subject lines, openings, or calls-to-action, you’re just hoping for the best. A/B testing lets you compare two versions of an email and see—based on real results—what gets more opens, clicks, or replies.
You don’t need to overcomplicate this. The goal is to get real data, improve your emails, and stop sending duds.
Step 1: Decide What to Test (and What Not To)
Before you open Scaledmail, pick one thing to test. Seriously, just one. If you change too much at once, you won’t know what made the difference.
Common things worth testing: - Subject line - First sentence/intro - Call-to-action (CTA) - Email length (short vs. detailed) - Tone (formal vs. casual)
What not to test (at least at first): - Fancy HTML templates (for cold email, plain text almost always works better) - Multiple variables at once (keep it simple) - Attachments (these can tank deliverability)
Pro tip: The subject line is the easiest place to start. If people aren’t opening your emails, nothing else matters.
Step 2: Write Your Variations
Write two versions: Version A (your “control,” or what you’d normally send), and Version B (the new version you want to test).
Example:
- Version A subject: “Quick question”
- Version B subject: “Can I ask you about [their industry]?”
Or if you’re testing intros:
- A: “My name’s Sam and I help companies like yours with X.”
- B: “I saw you’re working on Y—have you run into this problem?”
Don’t overthink it. Just make a single, clear change between A and B.
Step 3: Set Up the A/B Test in Scaledmail
Here’s how to do it in Scaledmail (assuming you’ve already signed up and added your contacts):
- Log in and create a new campaign.
-
Pick “Cold Email” as your campaign type.
-
Add your recipient list.
-
You can upload a CSV or paste in emails, depending on where your leads live.
-
Choose A/B Testing.
- Scaledmail offers a simple toggle or button for A/B testing (usually called “Split Test” or similar).
-
If you don’t see it, check their help docs—it’s not hidden, but it’s also not always labeled “A/B.”
-
Set up your variants.
- Paste Version A into “Email A.”
- Paste Version B into “Email B.”
-
Some platforms let you add more than two, but don’t get fancy. Two is enough.
-
Set the split.
-
50/50 by default is fine. If you have a small list, don’t bother with weird ratios.
-
Choose your sending schedule.
-
Avoid sending late at night or weekends unless you’re sure your audience works those hours.
-
Double-check your test.
- Send test emails to yourself for both versions.
- Look for typos, broken links, or weird formatting.
A note on list size:
If you’re testing on fewer than 100 people, don’t expect “statistical significance.” That’s fine; you’re looking for clear trends, not PhD-level proof.
Step 4: Launch the Test and Wait
Hit send. Now, don’t touch anything. Give it at least a few days (ideally a week) before you start reading into the results.
What to ignore while you wait: - One or two early replies (wait for enough data) - Your own anxiety about “not hearing back yet”
What to watch for: - Deliverability issues (are emails bouncing or landing in spam? If so, fix your list or your content) - Any weirdly high bounce or complaint rates (rare, but worth checking)
Step 5: Check the Results (What Matters, What Doesn’t)
Once enough emails have gone out, check the built-in analytics. Scaledmail shows you open rates, click rates, and reply rates for each version.
What really matters for cold email: - Reply rate: This is the real goal. Opens are nice, but replies pay the bills. - Open rate: Good for testing subject lines. If both versions are under 20%, you may have deliverability issues. - Click rate: Only matters if your CTA is a link (but generally, don’t ask cold leads to click links).
What doesn’t matter: - “Time spent reading” or “engagement scores”—these are distractions for cold email. - Minor differences (5% vs. 6% open rate). Look for meaningful gaps.
Pro tip: If there’s no clear winner, your test was probably too subtle, or your list is too small. Try a bigger change next time.
Step 6: Pick a Winner and Roll It Out (Or Try Again)
If Version B got a noticeably higher reply rate, congratulations—you’ve got a new “control” to beat next time. Move forward with it for your next batch.
If it’s a tie, or both versions flopped, don’t panic. This is normal. Cold email is tough, and most changes won’t produce magic results.
What to do next: - Test another variable (e.g., next time, try a different opening line) - Keep your tests simple and focused - Don’t chase “perfect.” Improvement is the goal.
Honest Takes: What Works, What’s Overrated
What actually moves the needle: - Short, clear subject lines - Personalization (using real, relevant details) - A call-to-action that’s easy to say yes to (“Do you handle this, or is it someone else?”) - Sending from a real person’s address, not a generic “info@”
What to ignore: - Fancy HTML or graphics (they hurt deliverability) - Mass personalization tokens (“Hi {FirstName}!” isn’t impressive if the rest is generic) - Chasing micro-optimizations (spend your time on bigger changes)
Common traps: - Testing too many things at once - Using tiny lists and expecting big insights - Letting “analysis paralysis” stop you from sending emails
A Few Real-World Tips
- Warm up new domains: If you’re using a fresh email address, warm it up before blasting out cold emails—or you’ll end up in spam.
- Rotate send times: If you send hundreds at once, stagger them. Scaledmail can handle this.
- Don’t obsess over open rates: Opens are tracked with images, and lots of people block those. Use reply rate as your north star.
- Keep your emails short: Nobody reads long cold emails. If you wouldn’t read it, they won’t.
Keep It Simple and Keep Testing
A/B testing in Scaledmail is straightforward if you don’t overthink it. Test one thing at a time, look at reply rates, and don’t chase tiny improvements. The best cold emailers are the ones who keep it simple, keep iterating, and don’t get discouraged when most things don’t work. That’s the real secret. Now go send something worth testing.