Domain Manage

Creating Drop Lists

Discussion in 'Drop catching Domain Names' started by danielcoates, Jan 16, 2016.

Thread Status:
Not open for further replies.
  1. danielcoates

    danielcoates Active Member

    Joined:
    Apr 2013
    Posts:
    132
    Likes Received:
    5
    I have a quick question about creating drop lists, according to the WHOIS terms I can't save any data from the database whatsoever, and the DAC terms state that I can hold a maximum of 1,000 records, for a maximum of 7 days.

    How can I store information about domain names, including there expiry date without breaching these terms?
     
  2. Domain Forum

    Acorn Domains Elite Member

    Joined:
    1999
    Messages:
    Many
    Likes Received:
    Lots
     
  3. Skinner

    Skinner Well-Known Member

    Joined:
    Jul 2008
    Posts:
    4,325
    Likes Received:
    81
    Its generally accepted you can store as much as data from the dac (but NOT whois) as your business needs.

    Search the forum for the 1,000 thing, as its more about non-relationship providing of data than what you can hold.
     
  4. danielcoates

    danielcoates Active Member

    Joined:
    Apr 2013
    Posts:
    132
    Likes Received:
    5
    Thanks, That's what I thought.

    I have searched the forum for every instance of the word 'DAC', I even used google with 'site:acorndomains.co.uk' filter, and I couldn't find anything.

    What do you mean non-relationship providing?
     
  5. Skinner

    Skinner Well-Known Member

    Joined:
    Jul 2008
    Posts:
    4,325
    Likes Received:
    81
  6. invincible

    invincible Well-Known Member

    Joined:
    Feb 2005
    Posts:
    4,108
    Likes Received:
    81
    The original poster raises a valid question and one that I have recently been thinking about myself. Unfortunately explaining it properly will cause 99.9% of readers eyes to glaze over.

    The Nominet DAC Terms & Conditions can be viewed here

    The relevant sections:

    5.2. Provided you do not breach any of the prohibitions and other terms of clause 5.3 or any other part of this contract you may do the following with the results generated by one or more DAC queries (in legal terms, we are granting you a non-exclusive, revocable licence to do these things, which we may terminate):

    5.2.1. compile a list of .uk domain names together with their expiry date or other data from the DAC query and (optionally) use that list for providing services to your existing clients"

    5.3. You may not do any or all of the following with information sourced from the register (whether from the DAC, PRSS and/or WHOIS), and whether newly received or stored under clause 5.2:

    5.3.6. knowingly provide more than an insubstantial part of the register or a sub-database of it (e.g. a database containing information released under the WHOIS or the DAC) to any one third party or end user in any 24 hour period. For the avoidance of doubt and as an example, more than 1000 registered domain names taken from the DAC would be more than an insubstantial part;

    5.3.7. retain (in any format) any information obtained from the DAC for a period of more than 7 days unless such information relates solely to domain names for which you are the registrant and/or registrar.
    _________

    I shall presume that the delayed DAC is more useful to assist with list making because the standard DAC does not contain return the "suspended [y|n]" field in output anymore. Knowing whether a domain name is suspended surely helps ascertain whether a domain name that is 1-92 days past its expiry date is likely to drop.

    I can accept that creating a master list of domain names alone can be achieved from sources available to the public.

    When a domain name is suspended it is removed from the zone file. It is possible to query the zone file using dig and one of Nominet's root name servers to establish whether domain names are within it. If no name servers are returned from the dig query the domain name queried has either (a) been suspended so cannot have name servers assigned (ignore what you see in WHOIS output; name servers may be displayed there for a domain name even if that domain name is suspended but the domain name will not be in the zone file) or (b) is not suspended but no name servers have been assigned to it.

    A list creator could decide to presume that the day name name servers are removed from a domain name is the beginning of its ~62 day suspension prior to the domain names deletion (I forget the exact figure but see this). They could then note the date that name servers were removed and continue to dig the domain name daily until 24 hours prior to the expected deletion of the domain name, monitoring to see if the domain name maintained no name servers (if the domain name was assigned name servers within that time frame it either must have been renewed or was never suspended in the first place). 24 hours before the presumed expiry date the list creator could finally check the domain name with the WHOIS or the DAC to ascertain whether it was (a) suspended and (b) to obtain its expiry date in order to validate the next day was 92 days after the date, hence the likely drop date.

    The problem with just relying on dig and querying the root name servers as above is that it is impossible to establish why a domain name has no name servers if they don't have any. It is not possible to establish either the suspension status or expiry date of domain names that have no name servers allocated just from the first dig query because some domain names will have had no name servers for periods of time that you will be unable to determine. You can use dig to establish a date when name servers are removed if you become aware of that date through continuous dig queries and that can be useful for deciding what to query via the delayed DAC.

    In order to seperate domain names that have no name servers into those that are definitely at least 30 days past their expiry date and are definitely suspended from those that are not the WHOIS or the delayed DAC would need to be queried and the data stored.

    Some domain names known as "special/out of sequence" suspensions do not delete 92 days after their expiration. It would not be possible to seperate those from the domain names that suspend at the usual ~62 days prior to expiry unless any domain names that had their name servers removed were immediately queried against the delayed DAC to establish the suspension [y|n] and expiry date status.

    The process seems likely to be close to the following:

    (a) Dig your master list of domain names and fork those that have no name servers to a new list. Dig the remainder of the master list and continually fork those that have no name servers to the new list.

    (B) Delay DAC the new list, and any new domain names added to it, and note the suspension status and expiry date of each domain name. Divide the output into the following further lists:

    i) All domain names that are suspended and are past expiry that follow the normal expiration path in relation to todays date. These are likely to delete 92 days after their expiry date. Calculate their expected deletion dates and store. Continue to dig this list daily to see if name servers are added as this indicates when domain names are no longer suspended and have probably been renewed. Delay DAC those that are due to be deleted 24 hours prior to their expected deletion dates and throughout the remaining hours to establish their absolute status. Some may be renewed at the last minute and some may have been renewed already but not had name servers assigned.

    ii) All domain names that are past expiry but not yet suspended. Add ~30 days to their expiry dates and store that these as the expected suspension dates. Continue to dig them each day to see if name servers are added which indicates they definitely cannot be suspended unless the name servers are removed again. Delay DAC domain names on their expected suspension dates to see if they have been suspended or renewed. Continue to dig those that are suspended daily to note if any are renewed. Delay DAC those that are suspended and haven't been renewed ~61 days (24 hours prior to expected deletion) after their expected suspension dates) to check whether they have been renewed but haven't had name servers assigned. If they haven't been renewed keep delay DACing them to keep up to date with their absolute status in case they are renewed at the last minute.

    iii) All domain names that are not past expiry or suspended but just have no name servers. Keep digging them to note if any name servers are ever assigned. Add ~30 days to their expiry dates and store that date as expected suspension date. Delay DAC domain names on their expected suspension dates to see if they have been suspended or renewed. Dig those that are suspended daily to note if any are renewed. Delay DAC those that are suspended and haven't been renewed ~61 days (24 hours prior to expected deletion) after their expected suspension date) to check whether they have been renewed without having name servers assigned. If they haven't been renewed keep DACing them to keep up to date with their absolute status in case they are renewed at the last minute.

    iv) All domain names that are suspended and don't follow the normal expiration path in relation to the current date. These could potentially be special/out of sequence deletions and will need to be monitored separately using the DAC because they could be deleted at any time. You may be able to establish a drop pattern for these in time.

    v) Any others, if there are any?

    It's possible that I have missed out some steps or not considered certain elements, doing the exercise from memory and not in practice. The way I have described is, as far as I can establish, the method to store the least data from the DACs and it may be possible to improve on it. Storing more data potentially contravenes 5.3.7.

    I'll be pleasantly surprised to receive an in depth technical response. :)
     
  7. danielcoates

    danielcoates Active Member

    Joined:
    Apr 2013
    Posts:
    132
    Likes Received:
    5
    Skinner: Thanks, I did skim through that post, I'll have a more in depth read after work tonight, thanks.

    ~

    lol, my immediate reaction was :shock:, but i persisted, and read the response.

    I doubt I could give you an in depth technical response, I'm not massively technically minded, and my mind kinda exploded and now aches a little, but I'll try and give a summary of what I understood in regards to my question.

    And to be honest, I didn't think I could query the ZONE file, I think I took it not being public, to mean it's not available to check. But now I take that to mean it's not available to download in it's entirety.


    ~ I can query DAC2 (Delayed DAC) and check the date the domain expires & suspension status, providing I don't store any information from the results for more than 7 days, giving me plenty of time to run other checks.

    ~ There would be nothing stopping me from storing a date that I calculated from information obtained from DAC2.

    ~ Before checking DAC2, I should check the ZONE file to see if it has name servers listed, and check DAC2 if it doesn't, scheduling further checks based on the results.

    So in conclusion, my database can contain the following, without breaching the rules:

    • Domain Name
    • Domain Length
    • Keywords
    • Google, Alexia & other useful information
    • Expected Drop Date, If any
    • Date I added it to the list
    • Date I last checked the Zone File
    • Date I last checked the DAC
    • Y/N on if it has Name Servers

    ~

    Thanks for the response, I have a lot of thinking to do on how I could implement this in PHP or Ruby, it's my intention to make an app to do this for me using cron jobs
     
    Last edited: Jan 16, 2016
  8. invincible

    invincible Well-Known Member

    Joined:
    Feb 2005
    Posts:
    4,108
    Likes Received:
    81
    :)

    Yes at the moment *.uk zone files cannot be downloaded in their entirety from Nominet and open zone transfers are not permitted either. Having them would allow you to dif them day by day to see what had changed.

    So say the rules.

    Possibly not as it isn't data directly taken from the output. To be sure you could check with Nominet, by email, if you felt you wished a firm answer.

    The dig command saves overusing the DAC because almost all domain names will be suspended so have their name servers removed before they are deleted, therefore you need to concentrate DAC querying on those that are not in the zone files. Continue to dig those that have name servers to monitor for changes. You may find some names are deleted "cold" (i.e. they had name servers, weren't suspended and were deleted from that status). Those may have been registrar deletions or other deletions due to some sort of contravention that did not invoke suspension first. You'll notice those using dig unless they are deleted and registered again, with name servers assigned, between your two digs. Obviously if some like these are deleted and registered without your knowledge you cannot help that.

    You should probably use a database. Whether you keep history is also up to you. Once you begin the exercise you may pick up on other points I haven't mentioned. Feel free to come back here and document them. :)
     
Thread Status:
Not open for further replies.

Share This Page