PDA

View Full Version : What is this???


CCR
10-23-2006, 04:32 PM
Yahoo! Slurp Spider, And why is it looking at user profiles?????? I don't thiink I like that!!!!!
siskiou
10-23-2006, 04:43 PM
The Slurp spider again??

I never quite understood what they are about and how do you know what they are looking at? :confused:
CCR
10-23-2006, 05:28 PM
The Slurp spider again??

I never quite understood what they are about and how do you know what they are looking at? :confused:

Go under "whose on line" and it says right there what there looking at, or for that matter, any of us.
Reefin'
10-23-2006, 06:07 PM
i believe thats how they update their search engine
CCR
10-23-2006, 06:10 PM
MSNBot Spider Viewing User Profile Is what's there right now
Ronjunior
10-24-2006, 08:54 AM
Whoever runs this site can create a file that tells spiders to not search and follow certain folders/links etc. Need to make a robots.txt file.
Here is what I have in a NASCAR site I run for the robots.txt file under the main public http folder.

User-Agent: *
Allow: /
Disallow: /cgi-bin

the * tells all search engines to not follow/search the cgi-bin. (can specify certain search engines if needing to, but why bother in this case, I don't want any in there)
You can add the Disallow: / command line for more folders you dont want to be searched, just add a line below the other dissalow like
Disallow: /cgi-bin
Disallow: /memberlist (or whatever folder is used specifically, etc., etc.)
impur
10-24-2006, 09:19 AM
The spiders scour the site and report back to the search engine. That way when someone searches for something we might have on our site it queries them to our site.
Nyles
10-24-2006, 10:07 AM
Its a plot to take over the world... there all watching! Accept Chief... LOL
Ahbrit
10-24-2006, 11:23 AM
You are correct Impur.
Ronjunior
10-24-2006, 07:30 PM
You are correct Impur.
Yes, Impure is correct, but some people are scared off with their info being collected. Maybe that is why people don't want to put what city they're from on the members info. If it's blocked as my suggestion above, that info wouldn't be collected, search engines see the universal command and have to follow it....or shall I say not follow it :) Personally, I could care less, I already have a website of my own out there, but am careful about what personal info I put on it. I can google ronjunior and this website and I get hits, but it doesn't bother me....I'm not hiding from the law, it's not worth it.(scary)