View Full Version : a lot of recurssion
blade700
07-20-2007, 09:53 AM
Is anyone using robots.txt to stop Google reading submit.php? There's no point in it trying to follow that link on every page is there?
I'm thinking that there's a lot fo recurssion at the moment and looking for ways to chop it down a bit.
vegabond
07-21-2007, 01:58 PM
No, I did not! If you don't want to be crawled by SE then you may set NO Follow attribute for this link :)
blade700
07-21-2007, 02:30 PM
thought the "nofollow" meant that it got traversed but no PR leak
droolingmnky
08-02-2007, 07:13 AM
The nofollow won't stop it from getting crawled, it'll just stop the passing of link juice. If you don't want it crawled then add it to robots.txt
vBulletin® v3.8.0, Copyright ©2000-2012, Jelsoft Enterprises Ltd.