Building a custom company database sounds like something only big tech firms bother with. But whether you’re doing sales prospecting, market research, or just want to stop wrestling with messy spreadsheets, scraping company data yourself can save a lot of headaches. This guide is for anyone who’s tired of unreliable databases and wants more control—especially if you’ve tried generic B2B data platforms and found the data outdated or full of holes.
Here’s the honest truth: building a database isn’t magic, but it does take some planning. We’ll walk through how to use the Proxycurl API to gather company data, make sense of it, and avoid common pitfalls. No fluff—just a clear, step-by-step process you can actually follow.
Step 1: Figure Out What You Actually Need
Before you sign up for anything or write a line of code, get clear on your goals. Here’s why: Proxycurl can grab a ton of company info, but if you try to collect everything, you’ll waste time (and probably money).
Ask yourself:
- What companies do I care about? (Industry, region, size, etc.)
- What data fields do I really need? (Name, website, employee count, funding, etc.)
- How often does this data need to be updated?
Pro tip: Start small. Build a proof of concept for 10-20 companies before you go all-in.
Step 2: Get Access to Proxycurl
Head over to Proxycurl and sign up. The free tier is limited, so if you’re planning to pull a lot of data, look at their pricing and decide what makes sense. Don’t just assume the free plan will cover a full project—API requests add up fast.
Things to watch out for:
- Proxycurl bills by API call—track your usage or you’ll get a surprise bill.
- Make sure you read their docs on rate limits and fair use. If you hammer the API, you might get blocked.
Step 3: Build Your Input List
Proxycurl can search by domain, company name, or LinkedIn URL. The more precise your input, the better your results.
Best options:
- LinkedIn Company URLs: Most accurate. If you already have these, use them.
- Company Website Domains: Pretty good, but can return false positives if the domain is generic.
- Company Names: Okay for big, unique names. For common names (“Acme Inc”), expect some junk.
How to get this info:
- Export from your CRM
- Scrape from LinkedIn (careful: this might break their terms of service)
- Buy a list (just make sure it’s legit)
Step 4: Pick the Right Proxycurl Endpoints
Proxycurl has a bunch of endpoints, but not all are worth your time. Here are the ones that matter for a company database:
- Company Profile Endpoint: Gets you the basics—name, description, website, size, location, industry.
- Company Employee Endpoint: Useful if you care about who works there.
- Company Search Endpoint: Lets you find companies by keyword, industry, location, etc.
Skip these (unless you have a weird use case):
- Deep enrichment endpoints (expensive, slow, often overkill for most databases)
- Anything that promises “real-time” updates—data is rarely truly fresh
Example API call (using Python’s requests):
python import requests
headers = { "Authorization": "Bearer YOUR_API_KEY" }
params = { "url": "https://www.linkedin.com/company/google/" }
response = requests.get( "https://nubela.co/proxycurl/api/linkedin/company/profile", params=params, headers=headers )
print(response.json())
Pro tip: Test with Postman or curl first. Don’t code up a big script until you’re sure you’re getting good data.
Step 5: Write a Script to Pull Data
You don’t need to be a software engineer, but a little Python goes a long way. Loop through your input list, call the API, and save the results. Here’s a bare-bones example:
python import requests import csv import time
API_KEY = 'YOUR_API_KEY' INPUT_FILE = 'companies.csv' OUTPUT_FILE = 'company_data.csv' RATE_LIMIT = 1 # seconds between requests
def fetch_company_data(linkedin_url): headers = {"Authorization": f"Bearer {API_KEY}"} params = {"url": linkedin_url} response = requests.get( "https://nubela.co/proxycurl/api/linkedin/company/profile", headers=headers, params=params ) if response.status_code == 200: return response.json() else: return None
with open(INPUT_FILE, newline='') as infile, open(OUTPUT_FILE, 'w', newline='') as outfile: reader = csv.DictReader(infile) fieldnames = ['input_url', 'name', 'website', 'industry', 'size', 'location'] writer = csv.DictWriter(outfile, fieldnames=fieldnames) writer.writeheader() for row in reader: data = fetch_company_data(row['linkedin_url']) if data: writer.writerow({ 'input_url': row['linkedin_url'], 'name': data.get('name', ''), 'website': data.get('website', ''), 'industry': data.get('industry', ''), 'size': data.get('size', ''), 'location': data.get('location', '') }) time.sleep(RATE_LIMIT)
What to watch out for:
- If you hit errors (429 or 403), you’re going too fast or you’re out of credits.
- API data isn’t always perfect—expect some fields to be missing.
Pro tip: Save the raw API responses too. You’ll thank yourself later if you need to debug or reprocess.
Step 6: Clean and Check Your Data
This is where most people drop the ball. API data is only as good as its source—expect typos, weird formatting, or missing info.
What to do:
- Run basic checks for blank fields, obviously wrong entries, or duplicates.
- Use simple spreadsheet tools or Python’s pandas to clean up.
- Don’t overcomplicate it. You can always refine later.
Reality check: No API gives you 100% current or accurate data, especially for smaller or fast-changing companies. If you need absolute accuracy, you’ll need to supplement with manual research.
Step 7: Store the Data Somewhere Useful
You don’t need a fancy database to start—CSV files, Google Sheets, or Airtable work fine for small projects. If you’re dealing with thousands of companies or need automated workflows, consider:
- SQLite or Postgres: Great for technical users.
- Airtable: Decent for non-coders, but can get expensive as you scale.
- Google Sheets: Fine for quick looks, but slow with big datasets.
Pro tip: Don’t waste time designing a complex schema until you know what you’ll actually use.
Step 8: Keep It Fresh (If You Need To)
Company data gets stale fast. If your use case depends on current info (like hiring signals or funding rounds), set a schedule to re-run your script.
- Monthly or quarterly updates are reasonable for most use cases.
- Watch your API usage so you don’t burn through credits on unnecessary refreshes.
If you only need a snapshot, don’t overthink it—just run your script when you need the data.
Step 9: Respect the Legal and Ethical Boundaries
Just because you can pull data from APIs doesn’t mean you should ignore terms of service. Proxycurl is designed to play nice with LinkedIn’s rules, but you’re still responsible for how you use the data.
- Don’t spam people or resell scraped data without checking the rules.
- If you’re in Europe or dealing with personal data, pay attention to GDPR.
Not legal advice, but don’t be reckless.
What Works, What Doesn’t, and What to Ignore
Works well:
- Enriching a list of companies with basic info (industry, size, website)
- Finding employee counts or pulling company descriptions
- Quick, one-off research projects
Doesn’t work so well:
- Getting up-to-the-minute info on startups or small businesses
- Pulling niche fields (like tech stack or obscure financials)
- Building a “set it and forget it” database—data always needs maintenance
Ignore:
- Fancy claims about “real-time” data or AI-powered enrichment. Most of this stuff is just API wrappers on top of LinkedIn or Crunchbase.
- Overengineering. You don’t need a microservices architecture for a list of companies.
Keep It Simple and Iterate
That’s the whole playbook. Don’t get bogged down in perfecting your process or chasing every last data point. Start with a small list, get the basics working, and improve as you go. The best company database is the one you’ll actually use—and the only way to know what you need is to try it.
If you get stuck, step back, simplify, and try again. There’s always another API call tomorrow.