Spiders looking in, leaving

[ballochr]

My site is about six weeks old, and has incoming links.

Spiders like Googlebot and MSN bot come regularly, look at the robots.txt file and leave again without ever spidering any pages.

The robots.txt file reads
User-Agent: *
Disalllow: /install/

The header template seems to be OK:
<meta name="robots" content="index,follow" />

I have enabled URL rewrite.

Could there be anything else wrong? PHPLD is the whole site - installed in the root.

[anon]

patience...

[David]

Can we see the url of you site?

[Optimit]

What version of phpLD?

[ballochr]

The site is ballochroy.com, version of phpLD is 3.03, thanks.

[Optimit]

Content visible to registered users only.

^^^There is the answer.

Version 3.03 of phpld was found to have some issues with getting indexed. A fix was put out, but upgrading to 3.04 is the best resolution.

[ballochr]

Thanks for the help! Will upgrade.

[Neticus]

Just a heads up..

I'm not sure but I think I've heard around the forum that 3.04 does not support the template you're currently using. Before you encounter the madness yourself.

[ballochr]

OK - I'll have to apply the fix for 3.03 then - can someone tell me where I can pick it up?

[mikedippel]

Mine looks like this:
_____________________________

User-agent: Mediapartners-Google*
Disallow:

# PARTIAL access (All Spiders)
User-agent: *
Disallow: /cgi-bin/
Disallow: /html/admin/
Disallow: /html/backup/
Disallow: /html/files/
Disallow: /html/images/
Disallow: /html/include/
Disallow: /html/install/
Disallow: /html/lang/
Disallow: /html/libs/
Disallow: /html/temp/
Disallow: /html/templates/
____________________________

I noticed that you spelled 'disallow' incorrectly. I also included a section for Google adsense that is supposed to assist in Google serving more appropriate ads.

[ballochr]

Michael -thanks for the correction.

Optimit - was the fix you mentioned to do with the headers not authenticating? I checked and indeed my site would not authenticate. I picked the fix up here
http://www.phplinkdirectory.com/forum/showthread.php?t=2850&highlight=rewrite

Hopefully the spiders will like my site now.

[Optimit]

Yes, that is one of the fixes. I will look and see what other fixes you may need. If you are able, you should join the supporters forum. Most of these patches and upgrades for 3.xx are available there.

[Steven Myers]

I had the same issue with the spiders visiting the robots.txt, come to find out, my txt was never filled, so I used michael's setup and it's working great so far :)

[ballochr]

Well, FYI the same day I applied the header authentication change Googlebot started spidering, and the day after that Inktomi started. This was after six weeks of wondering why nothing was happening, and buying high PR incoming links for the site.

Version 3.03 should come with a health warning:

"The production version of this software will not bring your site any search engine traffic."

If my host didn't provide good stats covering spider visits I'd still be in the dark.