If you’re sending cold emails and not sure what’s actually working, you’re not alone. Guesswork is the enemy of better results. This guide is for anyone using Reply’s outreach platform who wants to stop shooting in the dark and start running solid A/B tests. If you want practical steps, not marketing fluff, you’re in the right place.
Why A/B Test Your Outreach in Reply?
Let’s be blunt: Most email tweaks don’t move the needle. But sometimes a small change—like a subject line or call-to-action—can mean the difference between being ignored and getting replies. Reply’s A/B testing features make it easy to find out what actually works with your audience, not what some blog post told you should work.
Still, Reply’s A/B testing isn’t magic. It won’t fix a bad list, a weak offer, or spammy content. Think of it as a flashlight, not a silver bullet.
Step 1: Get Your Outreach Basics Right First
Before you even think about running tests, make sure:
- You’re emailing the right people. Testing copy on the wrong audience is useless.
- Your offer is clear and worth their time. No tweak can fix a bad pitch.
- Your list is clean. Invalid emails and spam traps ruin your results.
If you skip this, you’re just measuring which bad version is slightly less bad.
Step 2: Know What You Can Actually Test in Reply
Reply’s A/B testing is pretty straightforward. You can test:
- Subject lines
- Email body copy
- Call-to-action phrasing
- Timing (send times)
- Sequences/steps (if you want to get fancy)
What you can’t test: Attachments, rich media, or anything outside what Reply actually sends. Also, don’t expect it to magically bypass spam filters.
Step 3: Set Up Your First A/B Test
Here’s how to do it without overthinking:
-
Pick one thing to test.
Don’t change everything at once. If you tweak both the subject and the CTA, you won’t know which made the difference. -
Clone your step inside Reply.
- Go to your sequence.
- Click to add an A/B variant (“+ Variant” or similar, depending on UI updates).
-
Paste in your alternative copy for the subject line, intro, or CTA—whatever you’re testing.
-
Set your split.
Default is usually 50/50. Stick with that unless you have a reason to skew it. -
Save and activate.
That’s it. Reply will send each version to half your prospects.
Pro tip: Don’t test “Version A: Good Email” vs. “Version B: Terrible Email.” Make both versions solid, or you’re just confirming that bad emails don’t work.
Step 4: Let the Test Run (Don’t Jump at Shadows)
-
Give it time.
Resist the urge to call a winner after 10 sends. You’ll want at least 100-200 emails per version to see a real difference (more if your audience is diverse). -
Watch for randomness.
Sometimes Version B “wins” early, then flops. Don’t declare victory after a handful of replies. -
Keep an eye on deliverability.
If one version suddenly tanks on open rate, check if you tripped a spam filter.
Step 5: Analyze Results Honestly
Reply will show you:
- Open rates (Did they even see it?)
- Reply rates (Did they bother to respond?)
- Click rates (If you used a link—be careful, too many links = spam)
What matters most? Usually replies, not opens. Opens can be misleading (especially with privacy changes and pixel blockers). If all you care about is opens, you’re missing the point.
What’s “Statistically Significant”?
Most outreach teams don’t need to get fancy with math. If one version gets twice as many replies over a few hundred sends, that’s your winner. If it’s 10 vs. 11 replies, flip a coin—it’s not a real difference.
Step 6: Apply What You Learn—But Don’t Overreact
Let’s say Version B gets a noticeably higher reply rate. Great. Now:
- Make B your new “control.”
- Tweak something else next time. Maybe now try a different CTA or send time.
- Document what you tried. Otherwise, you’ll forget what actually worked (and what flopped).
Don’t get whiplash changing everything at once. Sometimes what “won” last month doesn’t work next month. People’s attention spans (and inboxes) change.
What Works, What Doesn’t, and What to Ignore
What Works
- Testing one thing at a time.
- Keeping copy short and clear.
- Using real language—not “just following up” fluff.
- Reviewing your actual replies—not just rates. Quality matters.
What Usually Flops
- Testing tiny changes (like a comma vs. a period).
You’ll never see a meaningful difference. - Obsessing over open rates.
Too many tools and privacy settings fudge these numbers. - Chasing trends.
What worked for a SaaS blog last year might not work for your audience.
Ignore This
- Vanity metrics: Who cares if you have a 70% open rate if no one replies or books a call?
- Templates that sound like templates: If you can tell it’s generic, so can your prospects.
- “Best time to send” myths: Test it for your list. Morning for one industry is dead time for another.
Pro Tips for Better A/B Testing in Reply
- Test bold ideas, not just safe tweaks.
Try a radically different subject or CTA once in a while. - Avoid spammy words.
“Free,” “guaranteed,” and “act now” get flagged. - Don’t overcomplicate.
If tracking and analysis take longer than writing the emails, you’re doing it wrong. - Archive your results.
A simple Google Sheet can save you from repeating dead ends.
Summary: Keep It Simple and Iterate
A/B testing in Reply is a tool, not a strategy. Use it to weed out what doesn’t work and double down on what does, but don’t fall for the trap of endless tinkering. The best outreach is clear, honest, and aimed at someone who actually wants to hear from you. Keep your tests simple, review your results, and don’t be afraid to try something new next time. That’s how you actually get better—one clear experiment at a time.