Jump to content

What is this???


Recommended Posts

Whoever runs this site can create a file that tells spiders to not search and follow certain folders/links etc. Need to make a robots.txt file.

Here is what I have in a NASCAR site I run for the robots.txt file under the main public http folder.


User-Agent: *

Allow: /

Disallow: /cgi-bin


the * tells all search engines to not follow/search the cgi-bin. (can specify certain search engines if needing to, but why bother in this case, I don't want any in there)

You can add the Disallow: / command line for more folders you dont want to be searched, just add a line below the other dissalow like

Disallow: /cgi-bin

Disallow: /memberlist (or whatever folder is used specifically, etc., etc.)

Link to comment
Share on other sites

You are correct Impur.

Yes, Impure is correct, but some people are scared off with their info being collected. Maybe that is why people don't want to put what city they're from on the members info. If it's blocked as my suggestion above, that info wouldn't be collected, search engines see the universal command and have to follow it....or shall I say not follow it :) Personally, I could care less, I already have a website of my own out there, but am careful about what personal info I put on it. I can google ronjunior and this website and I get hits, but it doesn't bother me....I'm not hiding from the law, it's not worth it.(scary)

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.


  • Create New...