Looking to squeeze more out of your email campaigns but tired of guessing what works? This guide is for anyone who wants real answers about their subject lines and email content—without the fluff. If you use email to reach customers, leads, or anyone you care about actually reading your messages, you need to know how to run a proper A/B test. We’ll walk through how to do it step by step in Mailgenius, so you can stop hoping and start knowing.
Why A/B Test Emails? (And Why Most People Get It Wrong)
If you're reading this, chances are you already know the basics: write a subject line, send an email, cross your fingers. Maybe you’ve even tried changing things up and hoped for better results. Here's the problem—without a clear test, you’re just making noise.
A/B testing is about running a controlled experiment. You send two (or more) versions of an email to similar groups, then see which one gets better results. This isn’t just about curiosity; it’s about avoiding lazy assumptions—because what you think will work often doesn’t.
Common mistakes to avoid: - Testing too many things at once (you won’t know what actually made a difference) - Not having enough people in each group (results are just noise) - Ending tests too early (patience, unfortunately, is required)
Step 1: Decide What to Test (And Don’t Overcomplicate It)
You can test almost anything: subject lines, preview text, sender name, email copy, images, call to action buttons, and so on. But here’s the truth: if you try to test everything at once, you’ll learn nothing.
Start with what matters most: - Subject lines: These get you the open. - Email content: This gets you the click or reply.
Pick one thing per test. For most people, start with subject lines—if your email never gets opened, nothing else matters.
Pro tip: If your open rate is below 20%, work on your subject lines first. If it’s already decent, experiment with your content.
Step 2: Set Up Your Variations Inside Mailgenius
Mailgenius makes it straightforward to set up A/B tests, but you still need to think before you click.
- Log in and create your campaign.
- Choose “A/B Test” (or “Split Test”) when you set up the email.
- Create your variations:
- For subject lines: Write two clear, distinct options. Don’t just add an emoji to one.
- For content: Change one thing at a time—headline, image, CTA, etc. If you change the whole email, you won’t know what worked.
- Keep everything else the same: Same list, same send time, same sender.
What to ignore: Don’t bother with micro-edits like switching a comma or changing “Hi” to “Hello.” Test big, obvious differences—otherwise, you’ll waste time chasing shadows.
Step 3: Define Your Audience and Split It Properly
A/B testing works when the groups are actually comparable. If you send Version A to your best customers and Version B to cold leads, your data’s garbage.
- Use Mailgenius’ random split tool to divide your list evenly and randomly.
- Aim for at least 100 recipients per version. Fewer than that, and your results are likely random luck.
- Exclude recent unsubscribers and inactive contacts. They’ll skew your numbers.
Pro tip: If your list is small, run the test across multiple campaigns and combine the results. It’s not perfect, but it’s better than pretending 12 opens mean anything.
Step 4: Launch Your Test and Wait (No, Really)
Hit send. Now, the hard part: do nothing for at least 24–48 hours (longer if your audience is slow to open emails). Don’t peek and declare a winner after 10 minutes—email opens trickle in.
- Set a clear end time for the test. Don’t move the goalposts.
- Monitor for delivery issues (bounces, spam complaints), but don’t obsess over minor differences in the first few hours.
What to ignore: Don’t stress over a 1% difference in open rates—statistical noise happens. Look for clear, meaningful improvements.
Step 5: Analyze the Results Honestly
After your test window is up, check your results in Mailgenius’ reporting dashboard.
- For subject lines: Look at open rates. A real win is usually 5%+ higher, not just a sliver.
- For content: Track click rates and replies, not just opens (since the subject line gets the open, but the content gets the action).
- Check for statistical significance. Mailgenius may flag this, but if not, use an online calculator. If you’re not sure, ask yourself: “If I re-ran this test, would I expect the same winner?”
If the result is unclear, don’t overthink it. Just pick one and move on, or rerun the test with bigger differences.
What to ignore: Don’t obsess over tiny wins. If both versions perform about the same, move on to a new idea.
Step 6: Roll Out the Winner and Keep Testing
Once you’ve got a winner, send future emails using that subject line or content (until it stops working—nothing lasts forever).
- Document what you tried and what worked. You’ll thank yourself later.
- Test again with new ideas. Email behavior changes over time—what works now may flop next quarter.
- Don’t fall in love with any one trick. People get bored, so keep experimenting.
What Works, What Doesn’t, and What to Ignore
What works: - Testing big changes (dramatically different subject lines or calls to action) - Keeping your test groups random and similar - Running tests long enough to get real results
What doesn’t: - Micro-changes (tweaking one word or punctuation—rarely moves the needle) - Ignoring statistical noise (don’t declare a winner based on a couple of opens) - Testing too many things at once
What to ignore: - Over-engineered subject lines that sound like clickbait (people see through it) - Fancy graphics or videos, unless you know your audience loves them - “Best time to send” myths—test your own list instead of following generic advice
Pro Tips for Better Results (And Less Headache)
- Document your tests: A simple spreadsheet is enough.
- Set realistic goals: A 100% open rate is a fantasy. Focus on consistent, small improvements.
- Keep your audience clean: Remove bounced emails, inactive accounts, and spam traps regularly.
- Don’t chase fads: If an “email guru” promises a magic formula, be skeptical. Test it yourself.
Keep It Simple—And Keep Going
A/B testing in Mailgenius isn’t rocket science, but it does take discipline. Focus on one thing at a time, look for real differences, and don’t get bogged down in tiny tweaks. Simple, honest experiments beat hype every time. Keep it simple, keep testing, and let your results—not your gut—decide what works.