I've been wondering about how to set the Robots.txt for my site for some time now and this has given me a clearer picture on what I maybe should or shouldn't be allowing Google to index.
Thank You and a Merry Christmoos from S.A.
Derrick Tan
· 2 years ago
Great post on the robotic thing.
Haha! Was always going to read stuffs on robot.txt but somehow just dropped the idea because of other stuffs.
I don't think it makes much of a difference. Shoe and John just disallow all those directories for security purposes...
Chris Jacobson
· 2 years ago
Is there any way to hide robots.txt from prying eyes and make it only viewable to search robots?
Clog Money
· 2 years ago
No it needs to be publicly accessible, you would be surprised how many robots actually look for this file, not just google but tons of other scrapers. Who knows you may even get the odd back link here and there from it.
Simon
· 2 years ago
I take the Shoemoney / Chow approach, as I only allow the indexing of posts themselves, and a couple of pages.
Shoemoney was given a few tips by Aaron Wall on how to get out of supplementals (when they were public), which mostly revolved around the robots.txt - it was to stop duplicate content being indexed, and according to Shoemoney:
"I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff."
I would expect that ProBlogger has been better organised from the start, and has never needed robots.txt to help with duplicate content. All his archives (categorys, dates etc) are just excerpts, which has always been thought to help.
Mike Huang
· 2 years ago
Milk Man, even though this is an interesting post...we all know you can do better. Where's the mojo? :)
-Mike
John Cow
· 2 years ago
Mojo, Mike?
Althaf
· 2 years ago
Good to see a useful post after a long time.
vhxn.com
· 2 years ago
Thanks for the interesting Post :arrow:
Think Like An SOB
· 2 years ago
lol. I have been using Shoe's robots.txt file for my blog since its inception. Figured, he probably knows what he's doing when it comes to SEO.
Nicholas James
· 2 years ago
Excellent post.
I completly forgot about robots.txt and this reminded me for my blog :wink:
Allyn Paul
· 1 year ago
This is way over my head! I have a plugin that creates a Google sitemap...does that suffice in terms of this posting and the robot txt?
John Cow
· 1 year ago
The sitemap will make it easier for Google to index your site. The robots.txt allows you to tell the search engine crawlers what they can and can't put on file.
Because of duplicate content that could be picked up by a crawler (a post in your archives has a different URL but still hods the exact same content as it's main URL) Google for example might think you're spamming. (although we're not convinced by that. Surely Google is smarter than that.)
ajacx
· 1 year ago
can i put this robot.txt in my blog spot?
Allyn Paul
· 1 year ago
Cow--thanks for taking some time to reply to my question...I appreciate the solid information! AL
allen johnson
· 1 year ago
Yah I would n,t blame you problogger is probably the best choice the reason why he doesn't hold anything from google is that he has so many incoming links that probably leads to his older post
All Comments
I've been wondering about how to set the Robots.txt for my site for some time now and this has given me a clearer picture on what I maybe should or shouldn't be allowing Google to index.
Thank You and a Merry Christmoos from S.A.
Haha! Was always going to read stuffs on robot.txt but somehow just dropped the idea because of other stuffs.
Will find out more on this.
All the best!
Regards,
Derrick Tan
http://www.learn-internet-marketing-free.com
Shoemoney was given a few tips by Aaron Wall on how to get out of supplementals (when they were public), which mostly revolved around the robots.txt - it was to stop duplicate content being indexed, and according to Shoemoney:
"I am happy to report not only am I out of supplemental hell but my Google traffic has increased 1400% in only 1 month after implementing his list of stuff."
I would expect that ProBlogger has been better organised from the start, and has never needed robots.txt to help with duplicate content. All his archives (categorys, dates etc) are just excerpts, which has always been thought to help.
-Mike
I completly forgot about robots.txt and this reminded me for my blog :wink:
Because of duplicate content that could be picked up by a crawler (a post in your archives has a different URL but still hods the exact same content as it's main URL) Google for example might think you're spamming. (although we're not convinced by that. Surely Google is smarter than that.)
AL