Large-Scale Real Estate Agent Data Collection for US Market Research

Cupid Real Estate provides residential real-estate services across Washington DC, Maryland, and Virginia, supporting clients with buying, selling, property search, and home valuation.

Operating across multiple local markets creates a strong need for reliable market and agent information. Structured data can support territory research, agent discovery, and broader analysis across different locations.

Real-estate agent data was distributed across multiple websites, each organized differently by state, city, county, or buying location. The geographic scale made manual collection impractical. HomeLight alone contained thousands of city inputs, while UpNest required traversal across thousands of city pages. Individual locations could also contain several pages of results.  

The websites introduced additional technical challenges. Dynamic elements required browser interaction; long-running crawls could encounter blocking or session issues, and similar agent information appeared under different field structures across sources. 

Key Issues 

• Thousands of cities, state, and county locations had to be processed systematically.  
• Dynamic pages, forms, pagination, and location controls required browser-aware collection.  
• Rate limits, blocking, and session changes could interrupt long-running crawls.  
• Agent fields differed between sources, making consistent data collection more difficult.  
• The same agent could appear across multiple locations, creating duplicate and data-quality concerns. 

A separate collection of workflows was developed for each source rather than forcing different websites through one generic crawler. Each implementation handled the structure and interaction model of its source while producing data that could be stored and processed outside the website.  

HomeLight City-Level Collection 

A Python and Selenium collector navigated HomeLight city pages, waited for agent cards, followed pagination, and captured agent profile information. The workflow collected fields such as name, brokerage, biography, experience, transactions, pricing information, city, and state.  

UpNest Geographic Crawling 

Scrapy was used to discover and process UpNest city pages at a larger scale. Zyte Smart Proxy supported the crawl where regional access and long-running collection could otherwise encounter restrictions. The extracted records included profile links, brokerage, experience, transactions, ratings, and location information.  

HomeOpenly Browser Automation 

A separate workflow used Crawlee and Playwright to automate HomeOpenly's buying request process. The active configuration worked through location-specific datasets and stored results separately. Python helpers were then used to combine selected JSON records and convert outputs into CSV format.  

Pagination & Long-Running Collection 

The collectors were designed around geographic traversal rather than one-time page extraction. Pagination handling allowed additional result pages to be processed, while location-based storage reduced the need to treat every run as one large uninterrupted collection job.  

Structured Data Processing 

Collected information was preserved in JSON and CSV formats for downstream processing. Source-specific outputs retained agent profile and performance information in structured form, providing a base for later cleaning, comparison, and reporting.  

Key Deliverables 

• Developed three source-specific real-estate data collection workflows.  
• Built Selenium-based HomeLight collection with city traversal and pagination.  
• Implemented Scrapy and Zyte-powered UpNest crawling for large geographic datasets.  
• Created Crawlee and Playwright automation for location-based HomeOpenly collection.  
• Added structured JSON/CSV output, dataset merging, and conversion utilities for downstream use.

Reduced Location-by-Location Research 

Agent information could be gathered automatically from configured locations instead of requiring researchers to visit directories and collect profiles individually. 

Created Structured Agent Datasets 

Profile, brokerage, experience, transaction, rating, and location information were captured in machine-readable files. This made the collected data easier to search, filter, and process. 

Improved Collection Continuity 

Pagination and location-based collection allowed larger crawls to be handled in smaller units. This reduced dependence on completing an entire geographic dataset within one uninterrupted session. 

Made Multi-Market Analysis More Practical 

Agent information from many locations could be reviewed as datasets rather than isolated website profiles. This provided a more useful foundation for comparing markets and identifying patterns across territories.