Building SaaS landing pages is half science, half guessing game. If you’re tired of arguing over button colors or headline copy, it’s time to run AB tests and let the numbers speak. This guide is for anyone in SaaS who wants to use Optimizely to get real answers—without getting lost in buzzwords or endless setup.
Here’s how to actually do it, step by step.
Step 1: Know What (and Why) You’re Testing
Before you even log into Optimizely, get clear on what you want to change and why. SaaS teams waste hours testing things that don’t matter.
- Pick a real goal. For most SaaS landing pages, that’s signups, demo requests, or trial starts. Don’t get distracted by “time on site” or “engagement” unless they truly tie to revenue.
- Choose a hypothesis. Example: “Changing the headline to mention our main benefit will increase demo requests by 10%.”
- Be specific. Vague ideas (“make the page better”) lead to useless tests.
Pro tip: If you can’t write down what you expect to happen, you’re not ready to test.
Step 2: Set Up Your Baseline Analytics
Optimizely can track conversions, but make sure your existing analytics (Google Analytics, Mixpanel, whatever) are working first. Otherwise, you’ll second-guess your results.
- Check that events fire. If your signup button isn’t tracked, fix that first.
- Know your baseline. What’s your current conversion rate? Write it down.
- Get your dev involved early if you need to add tracking or custom events.
Honest take: Analytics setup is boring, but skipping it means you’ll waste weeks—or worse, trust bad data.
Step 3: Create Your Optimizely Account and Project
If you haven’t already, sign up for Optimizely and set up a new project for your SaaS site.
- Name your project clearly. Don’t call it “Test 1.” Use something like “Acme SaaS Landing Page.”
- Pick the right platform. For most landing pages, you’ll use Optimizely Web (not Full Stack).
Heads up: Optimizely isn’t cheap. If you’re not already a customer, check if the price makes sense for your traffic and needs. There are cheaper/free tools if you’re just starting out.
Step 4: Add the Optimizely Snippet to Your Site
Optimizely runs by injecting a JavaScript snippet into your site’s <head>. This lets it control what visitors see.
- Get the snippet code from your Optimizely project.
- Paste it into your site’s global
<head>. For SaaS apps, this usually means your landing page template. - Deploy, then double-check the snippet loads on the page (use browser dev tools).
Don’t skip this: If the snippet loads too late (or not at all), your tests will flicker or fail. Speed matters.
Step 5: Build Your First Experiment
Now the fun part. In Optimizely, you’ll set up a new “experiment”—that’s their term for an AB test.
a. Choose Your Page and Variations
- Pick your landing page URL. If you have multiple, start with the one that gets the most traffic.
- Create your “Original” and at least one “Variation.” For example, change the headline, swap out a hero image, or tweak the signup button text.
- Keep variations minimal. One change per test is the gold standard. If you change everything at once, you won’t know what worked.
b. Use the Visual Editor or Code
- Visual Editor: Good for simple changes (text, images, buttons).
- Code Editor: Needed for bigger changes (layout, advanced interactions).
Don’t get fancy unless you need to. Simple tests are easier to read and trust.
Step 6: Set Up Your Audience Targeting
Not all visitors are equal. You might want to test only on desktop, or only on new users.
- Define your audience. Most SaaS pages should target “all visitors” unless you have a good reason not to.
- Segment if needed: By device, country, traffic source, etc.
Watch out: Over-segmentation kills your sample size. If you’re not sure, go broad.
Step 7: Set Up Your Goals (Conversions)
This is where testing often goes off the rails. If you measure the wrong thing, your whole test is pointless.
- Primary goal: Usually a signup, demo request, or whatever your real KPI is.
-
Secondary goals: Optional. Maybe you care about clicks on a pricing page, but don’t let this distract from your main goal.
-
How to set it up: In Optimizely, create a “Goal” and point it at a pageview (e.g., /thank-you) or a click event.
- Test it: Make sure the goal triggers when you complete the action yourself.
Don’t: Track “all clicks” or “page scrolls” as a main goal. That’s vanity data.
Step 8: QA Your Experiment
Testers love to skip this. Don’t.
- Preview your variations in Optimizely before launching.
- Run through the flow yourself in Incognito mode.
- Check for flicker or layout bugs—especially on mobile.
- Verify goal tracking fires in your analytics and in Optimizely.
If you have QA folks, now’s their time to shine. If not, rope in a coworker. Two sets of eyes catch more bugs.
Step 9: Launch Your Test
- Set your traffic allocation. 50/50 split is standard. Don’t get cute unless you have a reason.
- Start the experiment.
- Monitor in the first hour for any weirdness (broken layouts, traffic drop, etc.).
If there are major issues: Pause the test and fix them. Don’t let bad data pile up.
Step 10: Wait—and Don’t Peek Too Early
Here’s where most teams mess up. AB tests need time and traffic.
- How long? Depends on your traffic and conversion rate. A week is a bare minimum; 2-4 weeks is better.
- Use an AB test calculator to estimate sample size before starting (search “ab test calculator”—Optimizely has one, but so do plenty of others).
- Don’t stop early just because you see a jump in results. Early trends often reverse.
Pro tip: Picking winners too soon is the #1 reason tests “fail” in the real world.
Step 11: Analyze Your Results Honestly
When the test is done (enough traffic, enough time):
- Look for statistical significance. Optimizely will show you if a result is likely real, not just random chance.
- Check your analytics too. Sometimes Optimizely and Google Analytics disagree—usually because of ad blockers or tracking issues. If numbers are wildly different, dig in.
- Keep it simple. If your variation wins, great. If not, learn and move on.
Ignore: Tiny percentage wins (like 1-2%) unless you have massive traffic. That’s just noise.
Step 12: Ship the Winner (Or Try Again)
- If variation wins: Roll it out to 100% of users.
- If there’s no difference: That’s a result too. Move to a new hypothesis.
- Document what you learned. Save screenshots and notes for the team.
AB testing is about learning, not just “winning.”
What Actually Matters (and What Doesn’t)
Works: - Testing big changes (headlines, value props, layouts). - Focusing on your main KPI. - Running tests long enough for real data.
Doesn’t work: - Micro-changes (tweaking a single word) unless you have Amazon-level traffic. - Obsessing over “statistical significance” on bad data. - Running 10 tests at once with low traffic.
Ignore: - People promising “10x conversions” with secret tactics. - Most “best practice” lists. Your audience is unique.
Keep It Simple and Keep Testing
Don’t overthink it. Start with your most important page, test one meaningful change, give it enough time, and see what happens. Then do it again. AB testing in Optimizely isn’t magic, but it beats arguing over opinions.
Iterate, learn, and keep it honest. That’s how SaaS teams get better—one test at a time.