Bulk Domain Whois Query

[BR657]

Hi Guys, new poster, so go easy.

We have a requirement to do a bulk lookup of around 75,000 domain names to understand who owns them.

Is this something that is possible? And is anyone interested in helping me achieve it?

 

[Skinner]

Without a bunch of proxy's or IP's, its going to take best part of 3 month to scan them at 1,000 per day, any other way would breach Noms terms.

DAC can only return dates and tags.

DomainTools may have the historic whois or one of the other historic whois sites but will take time.

All long term stuff. I would download Domain Name Analyzer, or one of the other domain tools, load the 75k names and scan 1,000 per day to stay within nom's terms.

 

[Skinner]

Using Whois2 for this would breach nominets terms, and likely to lose the person their tag and membership.

I wouldn't recommend it, especially for a tag holder, but if you had an ISP which issues dynamic ip's you could use standard whois and disconnect and reconnect every 1,000 but Nominet systems spot this and will IP range block entire ISP's.

A few years ago my ISP was range blocked, when I contacted Nominet they told me this was why, because 1 member out of a million abused the system to everyone was blocked.

 

[AuthorityDomain]

I also have a very similar requirement to do a mass bulk whois for thousands of domains. I don't necessarily need the registrant details, just if domain is registered or not and creation / expiry dates. Especially as since GDPR won't be possible to get names anyway.

WHOIS2 according to nominet acceptable use policy states the following;
The maximum query rate is 100 queries per second with a maximum of 100,000 queries per rolling 24 hours. Therefore in theory this is possible. Only WHOIS(1) is 1000 queries per day.

If this the case, does anyone know of a service that would be able to do this? @Skinner

 

[Skinner]

You can't use all 100k on 1 person, whois2 is meant for registrar's to query the whois for clients.

You have to pass the IP for the cistomer, each customer is restricted to 1000 look ups, same as normal.

You could do what you need via DAC, a few 1000 look ups would take a matter of minutes.

I would use a program like Domain Name Analyser and just run 1,000 per day or per IP if your dynamic.

 

[AuthorityDomain]

Thanks for the answer Skinner. Makes sense.

I think DAC would be 1 solution if I had nominet membership and the code to do it! Think I'd rather pay someone or use a service!

100k names would take 100 days at 1000 a day! That's too long. Time is money

 

[Skinner]

I'm not sure how the rules with Nom would be on giving that much information out. The script to do such a thing is easy, but could land on do do with nominet. There are rules on how much data you can give out, perhaps @RobM knows more about how much you can give away.

 

[AuthorityDomain]

It seems someone like whoxy.com offers mass bulk whois - $200 for 125,000 lookups. I may be able to reduce the amount if I can do a bulk DNS check using WatchMyDomains Pro tool (similar to domain name analzer) to find all names with IP addresses. Those without IP's, I wont need to do whois check on.

 

[Edwin]

Dedupe against the zone files first? If you only really need to see if the domains exist or not that would cut the numbers down massively...

 

[Skinner]

That is a good idea, and not too difficult to do. Just need a function to query the zone and return null if not found, then write the domain to a table, then once the list is run dump table contents.

100k wouldnt take long at all.

 

[wb]

Following on from this, what's the simplest way these days using DAC to return the tags and expiry dates for a list of domains (either into a hosted database or local file)?

I previously used DAC Research Tool for Windows that @aZooZa kindly shared many years ago and worked perfectly... haven't needed to use it in a long time though and I'm on Mac now!

 

[Skinner]

WB, I don't think there is dedicated software (I haven't really looked), I haven't come across anything anyway, and usually you come across these things.

I posted 95% of the code, all you'd need to add would be your source of choice, file upload, input box, database query and your preferred output method and this will do it.

http://www.steven.co.uk/drop-catching/querying-the-dac-with-php/