[LLVMdev] llvm.org robots.txt prevents crawling by Google code search?

[Talin]

On Wed, Oct 13, 2010 at 11:10 PM, Anton Korobeynikov <
anton at korobeynikov.info> wrote:

> > indexing the llvm.org svn archive. This means that when you search for
> an
> > LLVM-related symbol in code search, you get one of the many (possibly
> > out-of-date) mirrors, rather than the up-to-date llvm.org version. This
> is
> > sad.
> This is intentional. The workload of the server was pretty huge w/o this.
>

Could we at least add a rule allowing the codesearch crawler, rather than
opening it up to all crawlers? The user agent string is
SVN/1.5.4/GoogleCodeSearch.

>
> --
> With best regards, Anton Korobeynikov
> Faculty of Mathematics and Mechanics, Saint Petersburg State University
>



-- 
-- Talin
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20101014/3b0e8851/attachment.html>