[llvm-dev] llvm bugzilla not indexable

[Jacob Lifshay via llvm-dev]

It probably could be setup to whitelist some of the well-behaved search
engine bots. If it's set to disallow by default then the broken bots should
not try to index it.

On Sun, Oct 7, 2018, 12:57 Anton Korobeynikov via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Hello
>
> We explicitly disallow indexing of bugzilla due to lots of bogus
> requests from broken bots which create major workload on llvm.org.
> On Sun, Oct 7, 2018 at 7:21 PM Ties Stuij via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Hi all,
> >
> >
> > As I was perusing LLVM Bugzilla, I noticed that it's not indexed by
> search engines, probably thanks to the robots.txt file* which inhibits all
> indexing.
> >
> >
> > I think that it's extremely useful for these issues to show up in say
> google search queries, so I wonder if it's just an oversight or if there's
> another reason for this.
> >
> >
> > Thanks,
> >
> > /Ties
> >
> >
> >
> > * http://bugs.llvm.org/robots.txt:
> >
> > User-agent: *
> > Allow: /index.cgi
> > Disallow: /
> >
> > IMPORTANT NOTICE: The contents of this email and any attachments are
> confidential and may also be privileged. If you are not the intended
> recipient, please notify the sender immediately and do not disclose the
> contents to any other person, use it for any purpose, or store or copy the
> information in any medium. Thank you.
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> With best regards, Anton Korobeynikov
> Department of Statistical Modelling, Saint Petersburg State University
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181007/22b66a7c/attachment.html>