Nobody likes to be caught off guard by a competitor’s surprise launch or pricing change. But who has time to manually check dozens of websites every week? If you’re running a business, in product, or in marketing, you know it pays to keep tabs on your competitors—but it’s a pain. The good news: you can automate a lot of this with tools like Bardeen, without learning how to code or paying for another overpriced SaaS.
Here’s how to set up a simple, no-nonsense system to scrape competitor websites and get alerts when something changes. No hype—just what works, what doesn’t, and how to get started.
Why Automate Competitor Monitoring?
Manual tracking is tedious and easy to forget. Automation means:
- You don’t miss changes (price drops, new products, big announcements)
- Your team stays in the loop, without you being the bottleneck
- You can focus on real work, not detective work
But here’s the truth: automation isn’t magic. You’ll need to spend a bit of time up front to set it up, and you’ll hit snags sometimes (scraping isn’t perfect). Still, it’s worth it if you want to stay sharp without burning hours.
What You’ll Need
- A free (or paid) Bardeen account
- The list of competitor pages you want to monitor
- A place to get alerts (email, Slack, etc.)
- A realistic idea of what’s possible—some sites are harder to scrape than others
Bardeen is a browser extension that lets you automate tasks by connecting your favorite tools. Think of it as Zapier, but with more flexibility for scraping and browser automation.
Step 1: Make a List of Competitor URLs
Don’t get fancy—just drop the URLs of the exact pages you care about into a Google Sheet or Notion table. Start with:
- Pricing pages
- Product announcement blogs
- Feature comparison tables
- Press release/news pages
Pro tip: Less is more. Monitoring everything is a recipe for alert fatigue. Pick the pages where changes actually matter.
Step 2: Install and Set Up Bardeen
- Go to Bardeen’s website and install the extension for Chrome or Edge.
- Sign up and log in.
- Pin the extension so you can access it quickly.
That’s it for setup. No desktop app, no long onboarding. If you’re already using Notion, Google Sheets, or Slack, connect those integrations now—it’ll save time later.
Step 3: Build a Scraping Playbook
Here’s where the magic happens. You’ll teach Bardeen what content to watch for changes. This isn’t as scary as it sounds.
A. Start a New Playbook
- Click the Bardeen extension icon.
- Hit “New Playbook”.
B. Add a Scraping Step
- Search for the action: “Scrape data from website”.
- Enter your competitor’s URL (start with one to test).
- Use the “Select elements” tool to highlight the part you care about (e.g., the price on a pricing page). Bardeen will record the CSS selector.
Tips: - Don’t select the whole page—pick the smallest section that matters, or you’ll get noise. - If the page uses JavaScript to load content, try to wait until it’s fully loaded before scraping. Sometimes you’ll need to tweak the settings or use the “Wait for element” option.
C. Extract the Data
- Test your selector by running the scrape.
- Make sure you’re getting just what you want (e.g., just the price, not the whole table).
If it’s not working, try: - Refreshing the page and re-selecting the element. - Right-clicking and “Inspect” to see if the content is actually in the page source (some sites hide stuff behind logins or anti-bot measures).
What doesn’t work:
Bardeen can’t scrape behind logins unless you’re logged in in your browser. It also struggles with heavy anti-bot protections or CAPTCHAs. If you hit those, you’ll need a more advanced tool or just monitor manually.
Step 4: Set Up Alerts
Now, decide how you want to get notified if something changes.
A. Choose Your Alert Channel
- Email: Simple, works for solo founders or small teams.
- Slack: Good for teams—set up a dedicated #competitor-alerts channel.
- Google Sheets/Notion: For logging changes over time (not real-time alerts, but useful for analysis).
B. Add the Alert Step in Bardeen
- After your scraping step, add an action:
- “Send email” or “Send Slack message”
- Or “Append to Google Sheet/Notion database” if you want a log
Customize the message to include the scraped data and a timestamp. Example:
“Acme Inc. just changed their pricing to $99/month. See page”
C. Only Alert on Changes
To avoid spam, have Bardeen compare the latest scraped data to the last value. Only send an alert if it’s different.
- Add a “Compare values” or “If changed, then...” step.
- Store the last value in a Sheet or Notion page, and check against it each run.
You’ll have to tinker a bit—sometimes the easiest way is to always log to a sheet, and then only alert if the last row is different from the previous one.
Step 5: Schedule It
Set your playbook to run on a schedule—daily or weekly is usually enough.
- In Bardeen, look for the “Scheduler” action.
- Set the frequency based on how often competitors make changes (if you monitor pricing, daily is fine; for blogs, maybe weekly).
Don’t run it every 10 minutes. That’s overkill, and you’ll probably get blocked or rate-limited by the website.
Step 6: Test and Refine
- Run the playbook manually a few times to make sure it works.
- Check that alerts look right and are only sent on real changes.
- If you get false positives (alerts for stuff you don’t care about), tighten up your selectors.
- If you get nothing, double-check the page hasn’t changed or moved.
What to ignore:
Don’t stress over minor formatting changes or social media widgets updating. Focus your scraping on content that actually signals a real update (prices, headlines, feature lists).
Pro Tips and Gotchas
- Sites will change: If a competitor redesigns their site, your scraping selector could break. Plan to check your setup every few weeks.
- Don’t annoy your team: Don’t blast everyone with every tiny change. Summarize weekly, or send only critical alerts.
- Respect robots.txt: Technically, scraping isn’t always “by the book.” Don’t hammer a site—be respectful and don’t abuse automation.
- Keep it simple: Start with just one or two key pages. Once that works, add more.
- Manual fallback: For really high-value pages with lots of anti-scraping tech, you may still need to check by hand. Don’t waste hours trying to automate the impossible.
Example: Monitoring a Competitor’s Pricing Page
Let’s say you want to watch acme.com/pricing for changes.
- Add the page to your Google Sheet.
- Use Bardeen’s “Scrape data from website” on the price element.
- Compare the scraped price to the last value in your Sheet.
- If it’s changed, send yourself (or your team) an email or Slack message.
- Schedule it to run at 9am every day.
That’s it—now you’ll know if they try to outmaneuver you on pricing, without having to check manually every morning.
Keep It Simple—Iterate as You Go
You don’t need a perfect solution on day one. Start monitoring a couple of pages, make sure the alerts are useful, and improve your playbook over time. If you hit a wall with scraping, don’t force it—sometimes a good old-fashioned Google Alert or manual check is still the best tool for the job.
Automating competitor monitoring with Bardeen won’t make you a business genius overnight, but it will save you time, keep you informed, and let you focus on work that actually moves the needle. Start small, tweak as you go, and don’t overthink it.