John Cow dot COM - Latest Comments in How Important Is Robots.txt For Google?

Re: How Important Is Robots.txt For Google?

allen johnson — Mon, 07 Jan 2008 11:39:56 -0000

Yah I would n,t blame you problogger is probably the best choice the reason why he doesn't hold anything from google is that he has so many incoming links that probably leads to his older post

Re: How Important Is Robots.txt For Google?

Allyn Paul — Tue, 25 Dec 2007 09:05:07 -0000

Cow--thanks for taking some time to reply to my question...I appreciate the solid information!
AL

Re: How Important Is Robots.txt For Google?

ajacx — Tue, 25 Dec 2007 07:41:01 -0000

can i put this robot.txt in my blog spot?

Re: How Important Is Robots.txt For Google?

John Cow — Mon, 24 Dec 2007 15:11:05 -0000

The sitemap will make it easier for Google to index your site. The robots.txt allows you to tell the search engine crawlers what they can and can't put on file.

Because of duplicate content that could be picked up by a crawler (a post in your archives has a different URL but still hods the exact same content as it's main URL) Google for example might think you're spamming. (although we're not convinced by that. Surely Google is smarter than that.)

Re: How Important Is Robots.txt For Google?

Allyn Paul — Mon, 24 Dec 2007 14:47:22 -0000

This is way over my head! I have a plugin that creates a Google sitemap...does that suffice in terms of this posting and the robot txt?

Re: How Important Is Robots.txt For Google?

Nicholas James — Sun, 23 Dec 2007 19:00:42 -0000

Excellent post.

I completly forgot about robots.txt and this reminded me for my blog :wink:

Re: How Important Is Robots.txt For Google?

Think Like An SOB — Sun, 23 Dec 2007 13:42:24 -0000

lol. I have been using Shoe's robots.txt file for my blog since its inception. Figured, he probably knows what he's doing when it comes to SEO.

Re: How Important Is Robots.txt For Google?

vhxn.com — Sun, 23 Dec 2007 13:12:55 -0000

Thanks for the interesting Post :arrow:

Re: How Important Is Robots.txt For Google?

Althaf — Sat, 22 Dec 2007 12:37:09 -0000

Good to see a useful post after a long time.

Re: How Important Is Robots.txt For Google?

John Cow — Sat, 22 Dec 2007 12:34:06 -0000

Mojo, Mike?

Re: How Important Is Robots.txt For Google?

Mike Huang — Sat, 22 Dec 2007 12:20:14 -0000

Milk Man, even though this is an interesting post...we all know you can do better. Where's the mojo? :)

-Mike

Re: How Important Is Robots.txt For Google?

Clog Money — Sat, 22 Dec 2007 09:37:11 -0000

No it needs to be publicly accessible, you would be surprised how many robots actually look for this file, not just google but tons of other scrapers. Who knows you may even get the odd back link here and there from it.

Re: How Important Is Robots.txt For Google?

Chris Jacobson — Sat, 22 Dec 2007 09:12:16 -0000

Is there any way to hide robots.txt from prying eyes and make it only viewable to search robots?

Re: How Important Is Robots.txt For Google?

Ruchir — Sat, 22 Dec 2007 09:10:44 -0000

I don't think it makes much of a difference. Shoe and John just disallow all those directories for security purposes...

Re: How Important Is Robots.txt For Google?

Derrick Tan — Sat, 22 Dec 2007 08:42:02 -0000

Great post on the robotic thing.

Haha! Was always going to read stuffs on robot.txt but somehow just dropped the idea because of other stuffs.

Will find out more on this.

All the best!

Regards,
Derrick Tan
http://www.learn-internet-m...

Re: How Important Is Robots.txt For Google?

Making Money in South Africa — Sat, 22 Dec 2007 08:24:19 -0000

Excellent Post!

I've been wondering about how to set the Robots.txt for my site for some time now and this has given me a clearer picture on what I maybe should or shouldn't be allowing Google to index.

Thank You and a Merry Christmoos from S.A.