[LLVMdev] llvm.org robots.txt prevents crawling by Google code search?

[Talin]

One of the tools I use most frequently when coding is Google codesearch.
Unfortunately, llvm.org's robots.txt appears to block all crawlers from
indexing the llvm.org svn archive. This means that when you search for an
LLVM-related symbol in code search, you get one of the many (possibly
out-of-date) mirrors, rather than the up-to-date llvm.org version. This is
sad.

For more info, see the codesearch FAQ entry (item 9):

    http://www.google.com/intl/en/help/faq_codesearch.html#regexp

-- 
-- Talin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20101013/cf5ec840/attachment.html>