Assistance with PHP DAC script required!

[timter51]

I've knocked together a PHP script which I'm using basically as a DAC checker. I'm opening up an fsockopen on the DAC, sending in an array of domains with fwrite, and outputting the returned strings using fgets (whilst using feof to check for end of file). I'm checking 3 domains a second to stay within limits.

The problem I'm having is that the script exits at random times for no reason, no matter how many domains are in the array. I could get 2000 domains into an array of 10000, and it'll just exit out. I can't work out what's going on... it's like the response from the DAC is cut short.

Can anyone with experience here take a quick look for me and lend a hand? It'll probably take a someone who knows what they're doing seconds to see where I'm going wrong! Please PM if you might be able to help.

 

[grantw]

Are you hammering it and getting blocked perhaps causing the script to timeout?

Sorry, just reread your post, not that then!

Grant

 

[Skinner]

I have noticed there is sometimes a discrepency with what the Dac Returns, I can't figure out where it comes from but randomly the dac returns what I can only describe a foreign response i.e. something unexpected. With my first scripts it resulted in an infinite loop, I solved it by only acting on N/Y/B.

The other thing to be aware of running out of is resources if you are limited ? PHP is a nightmare for overheads (memory wise) when dealing with large arrays.

Next one would be execution time. I set my execution time out to 600 seconds so when loading names I limit it to 14 * 60 * 600 names, could you be timing out ?

 

[monaghan]

If you are not looking for out and out speed, why have such a huge array? Wouldn't you be better off putting the names in a database table and pulling a name, stuff it through the DAC, update the database, pause and then loop? You can use your select statement to just pull the unprocessed records making it easy to restart if you crash out.

Also as has been suggested, if you are running in a hosted environment then you may well have timeout issues due to PHP settings.

 

[Aaron Clifford]

set_time_limit(0);

Stick that at the top of your PHP file, it may stop the timing out.

Also best to make sure that you are sleeping correctly, what code are you using to sleep with?

 

[timter51]

I have noticed there is sometimes a discrepency with what the Dac Returns, I can't figure out where it comes from but randomly the dac returns what I can only describe a foreign response i.e. something unexpected. With my first scripts it resulted in an infinite loop, I solved it by only acting on N/Y/B.

The other thing to be aware of running out of is resources if you are limited ? PHP is a nightmare for overheads (memory wise) when dealing with large arrays.

Next one would be execution time. I set my execution time out to 600 seconds so when loading names I limit it to 14 * 60 * 600 names, could you be timing out ?

Thanks Skinner, I've not noticed any dodgy responses yet but I'll look into that. You could right on overheads and execution time, though... I'm running this on a local WAMP installation, so I'll see how I have it configured for that.

If you are not looking for out and out speed, why have such a huge array? Wouldn't you be better off putting the names in a database table and pulling a name, stuff it through the DAC, update the database, pause and then loop? You can use your select statement to just pull the unprocessed records making it easy to restart if you crash out.

Also as has been suggested, if you are running in a hosted environment then you may well have timeout issues due to PHP settings.

Love the database idea, but my PHP skills are limited! It is something I want to get to eventually, I know this is how any serious domainer should be doing it. In the meantime, I have a desperate need for something that just works at my skill level.

set_time_limit(0);

Stick that at the top of your PHP file, it may stop the timing out.

Also best to make sure that you are sleeping correctly, what code are you using to sleep with?

Yes I've got that set_time_limit in, that was my first "aaah, I know what it is" fix, but no luck. I'm using usleep(400000); to sleep each iteration through the array, that seems to work fine as I'm not getting any blocks from the DAC.

 

[Skinner]

I use a triple database system, I'm sure there are better ways but my system was never meant to be this big was just a small personal database, so kinda stuck with it.

Database 1 is a hopper loaded with keywords and strings, basically its

id, keyword, scanned, date

Once I'm done catching for the day I load 600 seconds (5 x 600) worth of names where scanned = 0, running at 200ms intervals (roughly 5x per second, I chose this as it doesn't really mess up my rolling 24 hour totals, but can go as low as 70ms) on a 10 min cron.

I then separate the chaff from the wheat into 2 databases

Database 2 is a registered name database
Database 3 is available names.

With each iteration, I build an SQL query, and every 100th iteration I update the scanned database of those 100 names to scanned=1. I also run the mysql update when blocked or the loop kicks out.

I maintain the available names list to get idea's for domains and find useful FTR names too.

 

[monaghan]

Love the database idea, but my PHP skills are limited! It is something I want to get to eventually, I know this is how any serious domainer should be doing it. In the meantime, I have a desperate need for something that just works at my skill level.

Seems odd that as a self declared PHP noob that you find realtime network IO easier than databases, especially given that there is so much database documentation and very little network IO documentation

 

[Aaron Clifford]

Seems odd that as a self declared PHP noob that you find realtime network IO easier than databases, especially given that there is so much database documentation and very little network IO documentation

^ That! The database stuff is easy, all you probably need to get going is:

http://www.w3schools.com/php/php_mysql_select.asp
http://www.w3schools.com/php/php_mysql_update.asp

Then you could even use something like:

http://sanderkorvemaker.nl/test/flexigrid/

to view the database, or just use phpmyadmin.

 

[PaulGregory]

Thing is with DBs there are multiple ways of reaching the same goal, with different merits depending on what your next goal is. With network stuff, it's perhaps easier to be confident. And I dare say one good doc is better than 10,000 conflicting docs.

Speaking of conflicting advice...

set_time_limit(0);

Stick that at the top of your PHP file, it may stop the timing out.

Actually, although in the manual that's supposed to mean 'no limit' (so it would make logical sense at the start of the file), in real world use you may find you need to reset this again at the start of each loop so it'll reset the no-limit again and won't time out. (This I know from other PHP processing - I'm not yet using DAC but the comments here are fascinating).

 

[Aaron Clifford]

Where is my conflicting advice?

 

[Aaron Clifford]

To add to the above, I've used the timeout code on my own scripts that run for 24 hours a day without issue. So I was making my suggestion based on my own experience with the DAC.

 

[PaulGregory]

Sorry Aarron, I made a sloppy link between 'more documentation = conflicting advice' and my refinement of your suggestion. There is a minor conflict between "top of file" and "inside loop" but I mainly meant that my refinement contradicts the official part of the PHP documentation.

Not everyone's PHP is processed in the exact same way - I appreciate my contribution seems like voodoo PHP but it has worked for me when processing big arrays of product stock data (particularly on WAMP). If you are in an environment that sets a limit on an unlimited timelimit, resetting the timelimit regularly works. If you are in an environment that works as the docs suggest, it doesn't make a difference.

I appreciate this is not based on DAC experience, but it is based on WAMP experience and given that the OP appears to still have a timeout problem and is using WAMP I thought it was worth a mention.

 

[Aaron Clifford]

Ah right, I wasn't neigh saying what you said .