I’m trying to automate the process of sending many domain name lookup requests.
I tried to do this by sending for example http request to domain registry site like Whois.net, but I notice they block you after several attempts.
others like networksolutions adds token to the page to avoid automated requests.
Is there any ideas please?
I’m using Java.
2
You are currently going through middlemen (remote web servers) instead of querying the information more directly.
There’s a program on your host system called whois (which is what that whois.net site is named after!) which you can call as a subprocess, and the information out of it isn’t usually too hard to parse. Theoretically, you could write the client in Java directly, but why would you bother to duplicate all that effort?
1
While whois is often used as a method to determine domain name availability, it is not always real-time or accurate.
The best and proper way to do this would be to become an accredited registrar for the TLDs in question and use the registry’s APIs (which usually work with the extensible Provisioning Protocol (EPP)) to determine availability.
How easy and cheap this is would depend on how many TLDs need to be supported. Another cheaper option with some limits would be to become a reseller that has has an API and supports the TLDs in question.