Discussion in 'General Board' started by domaingenius, Feb 28, 2005.
Interesting project and idea to get your users to build your database of URLs and spider them for you using their resources not yours. You just dump the results into your system.
Can someone spider all .co.uk websites in the world and send them to me for my new search engine?
Yep.The reason I found it was because I noticed them indexing
many of my pages,
Separate names with a comma.