On mar, 2003-01-07 at 05:25, Erik Moeller wrote:
As of last weekend, all Wikipedia articles seem to
have disappeared from
the Google index. Wikipedia article URLs look like this:
http://www.wikipedia.org/wiki/<article-name>
We are not aware of an outage that might have caused the Google spider
to miss the pages. I would much appreciate it if you could shed some
light on the issue.
Upon noticing that the main pages and mailing list archives _are_
indexed, I have my suspicions about our robots.txt file; the line:
Disallow: /w
perhaps should be:
Disallow: /w/
The former may be accidentally blocking /wiki/<arcticle-name> paths
-- which of course form the bulk of our content! -- in addition to the
scripted pages via direct access to the /w subdirectory that it's
intended to block.
I have updated the robots.txt file; if indeed this is how the googlebot
was interpreting the line, I hope we can be respidered soon...
-- brion vibber (brion(a)pobox.com / brion(a)wikipedia.org)