Comments on: Robots.txt – Disallow a directory, but allow select files? http://www.lyndseo.com/2008/05/robotstxt-disallow-a-directory-but-allow-select-files/ Fri, 14 Oct 2011 17:42:31 +0000 hourly 1 http://wordpress.org/?v=3.1.3 By: nate http://www.lyndseo.com/2008/05/robotstxt-disallow-a-directory-but-allow-select-files/comment-page-1/#comment-12 nate Fri, 16 Jan 2009 02:03:41 +0000 http://www.lyndseo.com/blog/?p=103#comment-12 Robotstxt.org is way out of date with the features supported by the major engines. This is the best reference on the web: http://janeandrobot.com/post/Managing-Robots-Access-To-Your-Website.aspx (caveat, I work at Microsoft Live Search, and I help write this article). You could use the 'allow' directive, or build a creative pattern using the pattern matching. Make sure to test this using google's webmaster tools! nate Robotstxt.org is way out of date with the features supported by the major engines. This is the best reference on the web: http://janeandrobot.com/post/Managing-Robots-Access-To-Your-Website.aspx (caveat, I work at Microsoft Live Search, and I help write this article).

You could use the ‘allow’ directive, or build a creative pattern using the pattern matching.

Make sure to test this using google’s webmaster tools!

nate

]]> By: Michael VanDeMar http://www.lyndseo.com/2008/05/robotstxt-disallow-a-directory-but-allow-select-files/comment-page-1/#comment-11 Michael VanDeMar Fri, 23 May 2008 15:12:39 +0000 http://www.lyndseo.com/blog/?p=103#comment-11 Lyndsay, Unfortunately, no. Straight from <a href="http://www.robotstxt.org/robotstxt.html" rel="nofollow">Robotstxt.org</a>: <strong>To exclude all files except one</strong> This is currently a bit awkward, as there is no "Allow" field. The easy way is to put all files to be disallowed into a separate directory, say "stuff", and leave the one file in the level above this directory: <blockquote>User-agent: * Disallow: /~joe/stuff/ </blockquote> Alternatively you can explicitly disallow all disallowed pages: <blockquote>User-agent: * Disallow: /~joe/junk.html Disallow: /~joe/foo.html Disallow: /~joe/bar.html </blockquote> Sorry. :) Lyndsay,

Unfortunately, no. Straight from Robotstxt.org:

To exclude all files except one
This is currently a bit awkward, as there is no “Allow” field. The easy way is to put all files to be disallowed into a separate directory, say “stuff”, and leave the one file in the level above this directory:

User-agent: *
Disallow: /~joe/stuff/

Alternatively you can explicitly disallow all disallowed pages:

User-agent: *
Disallow: /~joe/junk.html
Disallow: /~joe/foo.html
Disallow: /~joe/bar.html

Sorry. :)

]]>