[llvm-dev] llvm bugzilla not indexable

[Anton Korobeynikov via llvm-dev]

Well... the reality is that the broken bots sometimes ignore
robots.txt (and therefore we have to block certain IP ranges) and
broken bots also mimic the benign ones.
On Sun, Oct 7, 2018 at 11:07 PM Jacob Lifshay <programmerjake at gmail.com> wrote:
>
> It probably could be setup to whitelist some of the well-behaved search engine bots. If it's set to disallow by default then the broken bots should not try to index it.
>
> On Sun, Oct 7, 2018, 12:57 Anton Korobeynikov via llvm-dev <llvm-dev at lists.llvm.org> wrote:
>>
>> Hello
>>
>> We explicitly disallow indexing of bugzilla due to lots of bogus
>> requests from broken bots which create major workload on llvm.org.
>> On Sun, Oct 7, 2018 at 7:21 PM Ties Stuij via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> >
>> > Hi all,
>> >
>> >
>> > As I was perusing LLVM Bugzilla, I noticed that it's not indexed by search engines, probably thanks to the robots.txt file* which inhibits all indexing.
>> >
>> >
>> > I think that it's extremely useful for these issues to show up in say google search queries, so I wonder if it's just an oversight or if there's another reason for this.
>> >
>> >
>> > Thanks,
>> >
>> > /Ties
>> >
>> >
>> >
>> > * http://bugs.llvm.org/robots.txt:
>> >
>> > User-agent: *
>> > Allow: /index.cgi
>> > Disallow: /
>> >
>> > IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you.
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>>
>> --
>> With best regards, Anton Korobeynikov
>> Department of Statistical Modelling, Saint Petersburg State University
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev



-- 
With best regards, Anton Korobeynikov
Department of Statistical Modelling, Saint Petersburg State University