getting dynamically generated directories spidered

[pugyrob]

Hi,
i have a few dynamically generated,database driven directories. I am wondering how to get these sites spidered as deeply as possible, or if there is a quick way to rewrite multi urls. The sites have around 11,000 pages.

Any advice appreciated

Thanks, Rob

 

[ScottJ]

You would need to get good quality links pointing deep inside the site Rob.

 

[DaveH]

From my experience URL rewriting would be an effective way along with deep links to certain pages.

EDIT:
Is the site running on Windows or Unix?

 

[mishmash]

Taking as an example a directory site from your signature:
http://financesites.co.uk/listings.php?view=new&page=2

At the time of writing for me there are only two results for the following search:
http://www.google.com/search?q=site:financesites.co.uk

1. Biggest issue in my view is the lack of content. Ideally this would be unique, fresh and regularly updated. Content can be generated in a number of ways, including automatically, for the above example it would be possible to include news and blog articles referencing the company. You're not going to get large numbers of pages indexed by, and staying in the Google index unless you've got either good original content or enough incoming links / traffic to justify that depth of indexing on your site.

2. There are no individual company pages. There isn't the depth of content there.

3. As previous posters have commented search engine friendly URLs would help - these directly do what you are asking increase the depth of Google indexing.

4. Ideally in a directory site you might virtually have a directory for each company eg.

http://financesites.co.uk/Sucden_UK_Ltd/
with a few pages under that eg.
http://financesites.co.uk/Sucden_UK_Ltd/Sucden_mortgages.html
http://financesites.co.uk/Sucden_UK_Ltd/Sucden_savings.html
http://financesites.co.uk/Sucden_UK_Ltd/Sucden_investments.html
You might also benefit from a page at:
http://financesites.co.uk/Sucden_UK_Ltd.html

5.Deep links coming in will help, but also important is rotating which deep pages you link to from your home page. If you do have a large directory and link to a few entries from your home page you generally see those linked from your home page gaining more traffic and a better search engine position. I like doing two things: a) linking to the most visited directory entries from the front page (perhaps reset every week/month) and b)linking to the latest entries reached via a Google search using the search terms as the anchor text. Both of these strategies build on success your site is already having.


URL rewriting can be achieved on both Unix and Windows webservers, though if you're on a shared hosting account you might be limited by your host as to what you can do.

I will PM the originating poster with a link to a case study where I have applied the above, and more, successfully.

 

[aquanuke]

As an example with http://financesites.co.uk/index.php?category=17 (one of your sites Rob)

in the .htaccess put something like

Options +FollowSymlinks
RewriteEngine On

#categorys
RewriteRule ^category-([a-zA-Z0-9_]*).* index.php?category=$1

Then change your directory script to something like index.php?category=$1 to something like category-$1-$2.html (you will need the correct variables)

Which would produce financesites.co.uk/category-17-Bonds-Advisories.html

 

[pugyrob]

Many thanks all,
Chris, thanks alot for that, thats just what im looking for.

Cheers

Rob