Looking to build a high-quality list of leads from LinkedIn but tired of doing it all by hand? This guide is for you. We’ll walk through how to scrape LinkedIn profiles using Scrapingbee, what you need to watch out for, and how to avoid the most common pitfalls. If you want practical, step-by-step advice—without the hype—you’re in the right place.
Why Bother Scraping LinkedIn for Leads?
Let’s be real: LinkedIn is the world’s biggest public Rolodex for professionals. If you’re in sales, recruiting, or just need to reach the right folks, scraping LinkedIn can get you the data you need—names, roles, companies—fast. But LinkedIn doesn’t exactly roll out the red carpet for scrapers, and their terms and bot detection are strict.
That’s where tools like Scrapingbee come in, but even then, this isn’t “push a button, get a goldmine.” You need to know what works, what doesn’t, and what’s going to get you in trouble.
Before You Start: Legal and Ethical Stuff
Let’s get this out of the way: scraping LinkedIn is a gray area at best. LinkedIn’s terms of service say you’re not allowed to scrape their site, and they’ve gone after people and companies for it. Is it illegal? That depends on where you live and how you use the data. If you’re scraping public profiles for internal research or B2B outreach, you’re probably flying under the radar, but don’t take that as legal advice.
Bottom line: - Don’t scrape private data or anything behind a login you don’t own. - Don’t go overboard—huge scraping jobs get flagged fast. - Use scraped data responsibly. Don’t spam people.
If you need 100% safety, buy leads or use LinkedIn’s official APIs (but those are limited and expensive).
Step 1: Figure Out What You Actually Need
Before you start writing code or setting up Scrapingbee, get clear on what you want:
- What info do you need? (Name, job title, company, email—good luck with that last one)
- How many leads are you after? (A few dozen? Thousands?)
- What kinds of profiles? (Industry, location, seniority, etc.)
Pro tip: The more focused your search, the less data you need to sift through, and the less likely you are to get blocked.
Step 2: Build Your List of LinkedIn Profile URLs
Scrapingbee needs a list of URLs to scrape. There are a few ways to get these:
- Manual Search: Use LinkedIn’s search, filter down, and copy profile URLs by hand. Slow, but safe.
- Google Search: Use Google with site:linkedin.com/in and keywords (e.g.,
site:linkedin.com/in "marketing manager" Seattle). Copy URLs from results. - Paid Tools: Services like Sales Navigator or tools that export search results (at your own risk).
If you’re scraping a lot, automate this part with a script—but remember, scraping search results is also against LinkedIn’s rules.
Step 3: Set Up Your Scrapingbee Account
Sign up at Scrapingbee and get your API key. Scrapingbee acts as a middleman: you send it a URL, and it fetches the page for you, handling things like headless browsers and proxies.
Why use Scrapingbee? - It handles browser rendering, so you get the real page content (not just the bare HTML). - It rotates proxies, which helps you avoid getting blocked. - It’s pay-as-you-go—no monthly surprises.
What to ignore: Don’t believe anyone who says Scrapingbee is “undetectable”—if you abuse it, LinkedIn will still catch on.
Step 4: Write a Script to Fetch LinkedIn Profiles
You can use any language, but Python is simple and well-supported. Here’s a basic example:
python import requests
API_KEY = 'your_scrapingbee_api_key' linkedin_urls = [ 'https://www.linkedin.com/in/example-profile-1/', 'https://www.linkedin.com/in/example-profile-2/', # ... more URLs ]
for url in linkedin_urls: response = requests.get( 'https://app.scrapingbee.com/api/v1/', params={ 'api_key': API_KEY, 'url': url, 'render_js': 'true', # Necessary for LinkedIn }, timeout=60 ) if response.status_code == 200: html = response.text # Save or parse the HTML here else: print(f"Failed to fetch {url}: {response.status_code}")
How it works: - Loops through your list of profile URLs. - Uses Scrapingbee’s API to fetch each one, with JavaScript rendered. - You get the full HTML back.
Watch out for: - Rate limits: Don’t hammer the API; add delays between requests. - Failed requests: Sometimes you’ll get errors or captchas—expect a 5-10% failure rate.
Step 5: Parse Out the Data You Need
Now you’ve got HTML for each profile. The next step is to extract the info you care about.
Use a library like BeautifulSoup (Python) to parse the HTML:
python from bs4 import BeautifulSoup
def parse_linkedin_profile(html): soup = BeautifulSoup(html, 'html.parser') name = soup.find('h1') title = soup.find('div', {'class': 'text-body-medium'}) company = soup.find('span', {'class': 'text-body-medium', 'dir': 'ltr'})
# This is just a rough start—LinkedIn changes their markup a lot.
return {
'name': name.get_text(strip=True) if name else '',
'title': title.get_text(strip=True) if title else '',
'company': company.get_text(strip=True) if company else ''
}
Honest take: LinkedIn’s HTML structure changes constantly. Don’t expect this code to work forever. Be ready to tweak your selectors every few months.
What to ignore: Anyone selling a “one-click” LinkedIn scraper. If it works now, it’ll break soon.
Step 6: Avoid Getting Blocked (and Other Headaches)
LinkedIn is aggressive about blocking bots. Here’s how to make your scraping last:
- Keep it slow: Wait 10-30 seconds between requests. Yes, really.
- Randomize headers: Scrapingbee lets you set custom headers—use them to look less like a bot.
- Handle captchas: If you see a lot of captcha pages in your results, you’re going too fast.
- Monitor errors: Don’t just run your scraper and hope. Look for spikes in failed requests or weird HTML.
Pro tip: Run small batches, check your results, and don’t be greedy.
Step 7: Clean and Store Your Leads
Once you’ve scraped and parsed the data:
- Clean it up: Remove duplicates, fix encoding issues, and standardize fields.
- Check for missing info: Not every profile will have everything you want.
- Store safely: Use a spreadsheet, database, or CRM—whatever fits your workflow.
What’s not worth it: Trying to scrape emails from LinkedIn. They’re almost never public, and scraping them is the fastest way to get flagged.
Step 8: Use Your Leads — Wisely
Don’t just blast cold emails at everyone you scraped. Take time to:
- Personalize outreach: Use the data to send relevant, targeted messages.
- Respect opt-outs: If someone says no, honor it.
- Stay human: If your outreach feels like spam, it probably is.
What Works, What Doesn’t, and What to Ignore
What works: - Scraping public profile info (names, roles, companies). - Keeping your scraping slow and targeted. - Regularly updating your scripts as LinkedIn changes things.
What doesn’t: - Scraping huge numbers of profiles in one go. - Trying to grab emails, phone numbers, or anything not public. - Relying on “magic” scraping tools that promise no maintenance.
Ignore: - Anyone promising “guaranteed undetectable scraping.” - Overly complex setups unless you really need them. - Leads lists that are months old—data goes stale fast.
Final Thoughts: Keep It Simple
Lead generation with Scrapingbee and LinkedIn scraping isn’t rocket science, but it’s not fire-and-forget, either. Start small, keep your process simple, and expect to tweak things as LinkedIn changes. Don’t overthink it, and don’t shoot for perfection—just focus on building a useful, up-to-date list. Iterate from there.
Happy scraping, and don’t do anything you wouldn’t want showing up on the front page of the internet.