Staying on top of what your competitors are doing online is one of the oldest tricks in the book. But let’s be honest—manually checking their websites is tedious, unreliable, and nobody has time for that. Whether you care about shifting prices, new features, or just want to know when they finally fix that typo in their hero banner, you need a system that works while you sleep.
This guide is for anyone who wants to automatically monitor competitor websites for changes—founders, product managers, marketers, or just the overly curious. We’ll walk through setting up a scraper with Zenrows, handling the real-world headaches scraping brings, and making sure you’re not wasting time chasing false alarms. No hand-waving, no magic. Just a practical way to keep tabs on the competition.
Why Monitor Competitor Websites?
Before we get into the weeds, let’s clarify why you’d even bother:
- Price tracking: Spot when your competitor drops (or hikes) their prices before your customers do.
- Product launches: See new features, SKUs, or content as soon as it’s live.
- Messaging: Notice shifts in positioning, new landing pages, or changes to the “About” page.
- Job postings: Infer company strategy or growth based on new roles.
Honestly, if you’re not doing this, you’re probably missing something your competitors already know about you.
Step 1: Decide What (and Where) to Monitor
Don’t try to boil the ocean. Figure out what actually matters:
- Homepage: Good for headlines, pricing banners, and big announcements.
- Pricing page: Obvious choice if your business is price-sensitive.
- Product pages: Watch for new products, descriptions, or feature tables.
- Blog/news: Track new posts for PR or content strategy clues.
- Careers: Not just for recruiters—hiring activity is a window into their plans.
Pro tip: Start with a single page that moves the needle for your team. You can always add more later.
Step 2: Get Set Up with Zenrows
Zenrows is a scraping API built to handle all the stuff that makes web scraping annoying: JavaScript rendering, anti-bot protection, and endless CAPTCHAs. It’s not magic, but it does save you from writing headless browser hacks or playing whack-a-mole with blocked IPs.
You’ll need:
- A Zenrows account (there’s a free tier if you want to poke around)
- An API key (from your Zenrows dashboard)
- Basic Python (or another language—examples here are in Python because it’s the path of least pain)
Step 3: Write a Script to Fetch and Parse the Page
Here’s the nutshell version:
- Use Zenrows to fetch the page (it handles dynamic content, so you’re not stuck with empty HTML).
- Parse out the part of the page you care about.
- Save the result somewhere.
Let’s do the bare minimum to make this work:
python import requests from bs4 import BeautifulSoup
ZENROWS_API_KEY = 'YOUR_API_KEY' COMPETITOR_URL = 'https://competitor.com/pricing'
def fetch_page(url): api_url = f'https://api.zenrows.com/v1/?apikey={ZENROWS_API_KEY}&url={url}&js_render=true' response = requests.get(api_url) response.raise_for_status() return response.text
def extract_pricing(html): soup = BeautifulSoup(html, 'html.parser') # Adjust selector to match the real pricing section pricing_section = soup.select_one('.pricing-table') return pricing_section.get_text(strip=True) if pricing_section else ''
if name == 'main': html = fetch_page(COMPETITOR_URL) pricing = extract_pricing(html) with open('last_pricing.txt', 'w') as f: f.write(pricing)
What’s happening here?
- We’re using Zenrows’ API to fetch the page, with JavaScript rendering turned on.
BeautifulSoup(an HTML parser) grabs the relevant section.- The output is saved to a file for later comparison.
Don’t overthink the selectors: Start with something simple (like .pricing-table), test it, and tweak as needed.
Step 4: Compare for Changes Over Time
Fetching the page once isn’t much use. The trick is to run this script on a schedule (daily, hourly, whatever matters to you) and check if anything’s changed.
Here’s a basic way to compare the current content to the previous run:
python import difflib
def compare_changes(old_text, new_text): diff = difflib.unified_diff( old_text.splitlines(), new_text.splitlines(), lineterm='' ) return '\n'.join(diff)
if name == 'main': # ... after fetching and extracting as above ... try: with open('last_pricing.txt', 'r') as f: old_pricing = f.read() except FileNotFoundError: old_pricing = '' diff = compare_changes(old_pricing, pricing) if diff: print('Change detected:') print(diff) # You could email yourself, send a Slack message, etc. else: print('No changes.') # Save the new version for next time with open('last_pricing.txt', 'w') as f: f.write(pricing)
What to actually do when something changes:
- Send yourself an email or a Slack notification.
- Log the change with a timestamp for future reference.
- Don’t get fancy at first—the point is to get notified, not to build a dashboard you never look at.
Step 5: Automate It—Scheduling Your Script
Nobody wants to run this by hand. Here are easy ways to automate:
- On Linux/macOS: Use
cron(crontab -e) to run your script daily or hourly. - On Windows: Use Task Scheduler.
- Or use GitHub Actions: If you want to keep code and logs in a repo.
Example cron entry (runs at 8am every day):
0 8 * * * /usr/bin/python3 /path/to/your/script.py
Heads up: If the site changes structure, your selector might break—build in some error logging so you don’t miss silent failures.
Step 6: Dealing with Real-World Headaches
Anti-bot measures: Zenrows gets you part of the way (headless browsing, anti-bot bypass), but no tool is perfect. Some sites are aggressive. If you get blocked:
- Try Zenrows’ advanced parameters (rotating proxies, custom headers).
- Slow down your scrape frequency. Once a day is usually safe.
- Scrape only what you need—don’t hammer the site.
Dynamic content: If you’re not seeing the right content, double-check with js_render=true (as above). If it’s still not working, the site might use more aggressive obfuscation or lazy loading.
Selectors breaking: Sites redesign often. Your .pricing-table might become #prices or just change structure entirely. When your script fails or gets empty results, check the HTML manually and update your selectors.
False positives: Tiny, invisible changes (like a date or tracking param) can trigger alerts. Filter out obviously irrelevant changes, or add some fuzziness to your diff logic if you’re getting too much noise.
Legal and ethical stuff: Only scrape public pages. Don’t try to bypass logins or scrape personal data. If your competitor has a robots.txt, at least read it (but note: it’s not a law, just a convention).
Step 7: Leveling Up (If You Need To)
Once you’ve got the basics working, here’s what you might want to add:
- Monitoring multiple pages: Loop through a list of URLs and selectors.
- Storing history: Save a timestamped log, not just the latest version, so you can see trends.
- Nice alerts: Integrate with Slack, Discord, or email for real-time notifications.
- Visual diff: For pages with lots of layout, look into tools that can screenshot and compare images, though this gets fiddly fast.
- Hosted monitoring: If you outgrow your scripts, Zenrows has integrations, or you can look at scraping orchestration tools—but don’t bother unless your life depends on it.
Don’t chase perfection: Most teams just need to know something changed, not to run a forensic analysis. Start simple.
What to Ignore
- Scraping every single page: You’ll drown in data and false alarms.
- Fancy dashboards: Unless you’ve got a team of analysts, this is overkill.
- Expensive enterprise tools: For most use cases, a script and a cron job get you 90% of the value.
Wrapping Up
There’s no silver bullet for monitoring competitor websites, but a simple Zenrows-powered script gets you most of the way there with minimal fuss. Start with one or two high-impact pages, keep your scripts tidy, and don’t get sucked into building a whole monitoring platform when a few lines of Python will do.
Iterate as you go. If something breaks, fix it. If you get overloaded with alerts, filter better. Above all, keep it simple—because the best monitoring system is the one you actually use.