CCR Posted October 23, 2006 Share Posted October 23, 2006 Yahoo! Slurp Spider, And why is it looking at user profiles?????? I don't thiink I like that!!!!! Quote Link to comment Share on other sites More sharing options...
siskiou Posted October 23, 2006 Share Posted October 23, 2006 The Slurp spider again?? I never quite understood what they are about and how do you know what they are looking at? Quote Link to comment Share on other sites More sharing options...
CCR Posted October 24, 2006 Author Share Posted October 24, 2006 The Slurp spider again?? I never quite understood what they are about and how do you know what they are looking at? Go under "whose on line" and it says right there what there looking at, or for that matter, any of us. Quote Link to comment Share on other sites More sharing options...
Reefin' Posted October 24, 2006 Share Posted October 24, 2006 i believe thats how they update their search engine Quote Link to comment Share on other sites More sharing options...
CCR Posted October 24, 2006 Author Share Posted October 24, 2006 MSNBot Spider Viewing User Profile Is what's there right now Quote Link to comment Share on other sites More sharing options...
Ronjunior Posted October 24, 2006 Share Posted October 24, 2006 Whoever runs this site can create a file that tells spiders to not search and follow certain folders/links etc. Need to make a robots.txt file. Here is what I have in a NASCAR site I run for the robots.txt file under the main public http folder. User-Agent: * Allow: / Disallow: /cgi-bin the * tells all search engines to not follow/search the cgi-bin. (can specify certain search engines if needing to, but why bother in this case, I don't want any in there) You can add the Disallow: / command line for more folders you dont want to be searched, just add a line below the other dissalow like Disallow: /cgi-bin Disallow: /memberlist (or whatever folder is used specifically, etc., etc.) Quote Link to comment Share on other sites More sharing options...
impur Posted October 24, 2006 Share Posted October 24, 2006 The spiders scour the site and report back to the search engine. That way when someone searches for something we might have on our site it queries them to our site. Quote Link to comment Share on other sites More sharing options...
Nyles Posted October 24, 2006 Share Posted October 24, 2006 Its a plot to take over the world... there all watching! Accept Chief... LOL Quote Link to comment Share on other sites More sharing options...
Ahbrit Posted October 24, 2006 Share Posted October 24, 2006 You are correct Impur. Quote Link to comment Share on other sites More sharing options...
Ronjunior Posted October 25, 2006 Share Posted October 25, 2006 You are correct Impur. Yes, Impure is correct, but some people are scared off with their info being collected. Maybe that is why people don't want to put what city they're from on the members info. If it's blocked as my suggestion above, that info wouldn't be collected, search engines see the universal command and have to follow it....or shall I say not follow it Personally, I could care less, I already have a website of my own out there, but am careful about what personal info I put on it. I can google ronjunior and this website and I get hits, but it doesn't bother me....I'm not hiding from the law, it's not worth it.(scary) Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.