How to extract structured data from dynamic web pages using Zenrows API

If you’ve ever tried scraping data from a modern website, you know it’s not the 2000s anymore. Static HTML is rare, and most sites now load everything with JavaScript. Your usual tools—requests, BeautifulSoup—don’t cut it. This guide is for anyone who needs to pull structured data (like product listings, reviews, or prices) from dynamic web pages, and is tired of running into “content not found” errors. We’ll walk through getting it done with the Zenrows API, which takes the pain out of scraping JavaScript-heavy sites.

No fluff, no sales pitch—just what works, what doesn’t, and how to get usable data without pulling your hair out.

Why dynamic pages are a pain

Let’s get this out of the way: scraping dynamic pages is hard because:

  • Content is loaded by JavaScript, sometimes after the initial page load.
  • HTML might look empty, or just have loading spinners, if you fetch it the usual way.
  • Sites use anti-scraping tricks (CAPTCHAs, IP blocks, weird HTTP headers).
  • The markup changes often, so brittle scrapers break a lot.

If you’ve tried to scrape with just Python requests, you’ve probably gotten back a blank page or a mess of script tags—useless. Headless browsers (like Puppeteer or Selenium) work, but they’re slow, eat up memory, and are a pain to manage at scale.

This is where Zenrows comes in: it gives you a simple API endpoint that fetches the page, runs the JavaScript, and returns the actual content—without needing a local browser or a rotating proxy army.

Step 1: Sign up and get your Zenrows API key

First things first—sign up for a Zenrows account and grab your API key from the dashboard. You’ll need this for every request. There’s a free tier, but if you’re scraping a lot, expect to hit limits. (Pro tip: Don’t burn through your quota on test runs—save it for real data pulls.)

Step 2: Figure out what data you want

Before you write any code, inspect the site and decide:

  • What exact data do you need (product titles, prices, ratings, etc.)?
  • Where does it live in the rendered HTML or the API calls?
  • Are there anti-bot protections (rate limits, CAPTCHAs)?

Use your browser’s Developer Tools (F12) to:

  • See if the data appears after a delay (watch the Network tab).
  • Check if the data loads via a background API call (sometimes you can scrape that endpoint directly, which is faster and more reliable).
  • Copy the CSS selectors or XPath paths to your target data.

Honest take: If the site loads everything via a public API call in the background, you might not need Zenrows. Just hit that endpoint directly. But if the data only shows up after JavaScript runs, keep reading.

Step 3: Make a test request to Zenrows

Here’s the basic structure:

bash curl "https://api.zenrows.com/v1/?apikey=YOUR_API_KEY&url=https://example.com"

Let’s break that down:

  • apikey: Your API key from Zenrows.
  • url: The web page you want to scrape.

Zenrows will return the fully rendered HTML—JavaScript and all. Here’s a Python example using requests:

python import requests

ZENROWS_API_KEY = 'YOUR_API_KEY' target_url = 'https://example.com'

response = requests.get( 'https://api.zenrows.com/v1/', params={ 'apikey': ZENROWS_API_KEY, 'url': target_url, } ) html = response.text print(html[:1000]) # Print the start of the HTML so you can check if it worked

Reality check: Sometimes, Zenrows can get blocked if the site is very aggressive about anti-bot measures. If you see a CAPTCHA or a login wall, you’ll need to tweak your settings (see next step).

Step 4: Handle anti-bot measures (if you hit a wall)

Some sites don’t go down without a fight. If you’re seeing CAPTCHA pages or missing content, try these options:

  • Headless browser rendering: Zenrows can use a real browser under the hood (js_render=true). This handles more complex JavaScript.
  • Premium proxies: Some sites block traffic from known data centers. Zenrows lets you use residential proxies (premium_proxy=true), which look like real user IPs.
  • Custom headers: Mimic a real browser by setting the User-Agent or other headers.

Example with extra options:

python response = requests.get( 'https://api.zenrows.com/v1/', params={ 'apikey': ZENROWS_API_KEY, 'url': target_url, 'js_render': 'true', 'premium_proxy': 'true', 'headers': '{"User-Agent": "Mozilla/5.0"}' } )

Pro tip: Don’t enable every option by default. They cost more and slow things down. Start simple and only escalate if you’re blocked.

Step 5: Parse the structured data

Now you’ve got a big HTML blob—but you probably just want the data, not the markup. There are a few ways to extract structured data:

1. Use Zenrows' built-in extraction (recommended for standard patterns)

Zenrows can extract common data types (products, articles, reviews) automatically. You just set the auto_parsing=true parameter:

python response = requests.get( 'https://api.zenrows.com/v1/', params={ 'apikey': ZENROWS_API_KEY, 'url': target_url, 'auto_parsing': 'true' } ) data = response.json() print(data)

This works great if the site follows standard markup (like schema.org microdata or JSON-LD). But if the site is messy or custom, results may be hit-or-miss.

2. Extract data yourself (more control, more work)

If auto-parsing doesn’t cut it, you’ll need to parse the HTML yourself. Use Python’s BeautifulSoup or similar tools:

python from bs4 import BeautifulSoup

soup = BeautifulSoup(html, 'html.parser') for product in soup.select('.product-card'): title = product.select_one('.product-title').get_text(strip=True) price = product.select_one('.price').get_text(strip=True) print(title, price)

Honest take: The markup on modern sites is often a moving target. Expect your selectors to break occasionally. Keep your scrapers simple and fix them when they break—don’t chase “perfect” code.

Step 6: Iterate and handle edge cases

  • Test with different pages (pagination, filters, etc.).
  • Watch for “load more” buttons or infinite scroll—sometimes you’ll need to simulate these by passing the right URL parameters.
  • Save the raw HTML the first few times so you can debug with real data.

Ignore: Don’t waste time on fancy headless browser frameworks unless you really need them. Zenrows handles most dynamic content out of the box, and you can always fall back to manual debugging if something slips through.

What works, what doesn’t, and when to move on

What works: - Zenrows is great for sites that rely heavily on JavaScript but don’t have extreme anti-bot defenses. - The API is simple—no infrastructure to maintain. - Auto-parsing is handy for well-marked-up sites.

What doesn’t: - If the site uses heavy CAPTCHAs or requires a login, scraping is hard (sometimes not worth the effort). - Auto-parsing struggles with messy or custom HTML. - Using every advanced option (premium proxies, JS rendering) gets expensive fast.

What to ignore: - Don’t overengineer. You don’t need a distributed scraping farm for most use cases. - Don’t try to scrape sites that explicitly block all bots if you can get the data elsewhere (APIs, exports, partnerships).

Wrap-up: Keep it simple, keep it honest

Scraping dynamic web pages is always a bit of an arms race. Zenrows removes a lot of the busywork, but it’s not magic. Start with the basics, test as you go, and don’t be afraid to adjust your approach if a site changes. Most of the time, simple tools and a bit of patience will get you what you need—no heroics required.

If you hit a wall, step back and ask: do I really need this data, or is there an easier way? Don’t let scraping eat your whole week. Automate the boring stuff, but don’t get lost in the weeds.