Quantcast
Channel: ScrapeBox Forum - All Forums
Viewing all articles
Browse latest Browse all 5962

Handling large list for "Checking Unregistered domains"

$
0
0
Hey guys,

Straight to the problem: I have a 3 millions list domain to check for availability. I usually use "Check for Unregistered domains" with smaller list, up to 200K domains and it works great.

This "plugin" works in 2 pass:
-/The DNS pass (Checking if the domain has a DNS zone attached to it, in which case it's not available) goes real quick.
-/The whois pass, to actually check if the domain as a registrant attached to it, keeps getting slower. Starting from 50 threads, and once your proxies get banned from checking whois (temporary ban), the threads goes down to 1 active only and never goes up again (it should doesnt it?).

-> Working with not so much domain was easy, but now that I'm facing about 200k domain in the whois pass itself, I wonder how I can improve the active threads, more specifically, how can i reopen threads once the temporary ban on IPs has been gone.

Any suggestions is welcome.

Viewing all articles
Browse latest Browse all 5962

Trending Articles