[llvm-dev] PIC and mcmodel=large on x86 doesn't use any relocations

[John Reagan via llvm-dev]

To follow up on this...  I had a few conversations at the developer's meeting
about this.  I can see that "large" is really a big hammer for what I want to
do.  I'm going to relook to see if "medium" is a better fit although I think
I might want a "medium+" setting since each segment will fit into 2GB (ie
code, GOT, PLT, etc) but the initial distance between my GOT and code might
be larger and 2GB (this is the part I need to double check on what we
currently do on OpenVMS Itanium).  

As for the current behavior of asking for "-fPIC -mcmodel=large" and getting
them ignored without any warning, I find that unfriendly.  

Where would be the best place to filter that combination and issue some
warning?  And remember, such settings might be coming from a non-Clang
frontend...  I'm looking for a good starter bug for myself to dip my toes
into a real commit and this looks like a place to start.

John

> Date: Tue, 1 Nov 2016 10:06:44 -0400
> From: John Reagan via llvm-dev <llvm-dev at lists.llvm.org>
> To: <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] PIC and mcmodel=large on x86 doesn't use any
> 	relocations
> Message-ID: <00c301d23449$2ddd23d0$89976b70$@net>
> Content-Type: text/plain;	charset="utf-8"
> 
> 
> > From: Joerg Sonnenberger via llvm-dev <llvm-dev at lists.llvm.org>
> > To: llvm-dev at lists.llvm.org
> > Subject: Re: [llvm-dev] PIC and mcmodel=large on x86 doesn't use any
> > 	relocations
> 
> >
> > On Mon, Oct 31, 2016 at 02:30:43PM -0400, John Reagan via llvm-dev
> > wrote:
> > > The idea to make this all work on x86-64 is to use the address of a
> > > routine's PLT as its "address" in all cases.  By putting the PLT in
> > > 32-bit address space, some ancient BASIC or Fortran program can
> > > continue to use an INTEGER*4 to hold a routine's "address" and pass
> > it
> > > around.  Of course, our PLTs won't be actual dynamic loading code
> > > but just trampolines to the 64-bit code.  The PLTs provide a
> > > consistent "address of routine" for pointer comparisons, etc just
> > > like the "primary function descriptor" on Itanium.
> >
> > That's effectively how the main binary works for dynamically linked
> > non-PIE x86_64 code in the default code model. It creates some
> > problems for protected symbols and involves the nasty copy
> > relocations, but nothing so far seems to require the large code
> model,
> > at least in the sense of "GOT and PLT are not addressable with 32bit
> signed offset"?
> >
> > Joerg
> >
> 
> Looking at what I have today on OpenVMS Itanium, I can have code at
> address (I'll add a 'dot' in the middle for readability)
> 
> 0x00000001.00000000
> 
> and our short section with static data/FDs is at
> 
> 0x00000000.00020000
> 
> How does the code find the GOT/PLT with only 32-bit %rip addressing?
> 
> 
> 
> 
> ------------------------------
> 
> Message: 2
> Date: Tue, 1 Nov 2016 10:14:13 -0400
> From: John Reagan via llvm-dev <llvm-dev at lists.llvm.org>
> To: <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] What was the IR made for precisely?
> Message-ID: <00c701d2344a$39202d40$ab6087c0$@net>
> Content-Type: text/plain;	charset="utf-8"
> 
> > Date: Tue, 1 Nov 2016 11:31:05 +0000
> > From: David Chisnall via llvm-dev <llvm-dev at lists.llvm.org>
> > Subject: Re: [llvm-dev] What was the IR made for precisely?
> >
> > On 28 Oct 2016, at 21:25, Hal Finkel <hfinkel at anl.gov> wrote:
> > >
> > > ----- Original Message -----
> > >> From: "Chris Lattner via llvm-dev" <llvm-dev at lists.llvm.org>
> > >> To: "David Chisnall" <David.Chisnall at cl.cam.ac.uk>
> > >> Cc: llvm-dev at lists.llvm.org, "ジョウェットジェームス"
> > <b3i4zz1gu1 at docomo.ne.jp>
> > >> Sent: Friday, October 28, 2016 2:13:06 PM
> > >> Subject: Re: [llvm-dev] What was the IR made for precisely?
> > >>
> > >>
> > >>> On Oct 28, 2016, at 1:21 AM, David Chisnall
> > >>> <David.Chisnall at cl.cam.ac.uk> wrote:
> > >>>
> > >>> On 28 Oct 2016, at 02:43, ジョウェットジェームス
> > <b3i4zz1gu1 at docomo.ne.jp>
> > >>> wrote:
> > >>>>
> > >>>> I would need to sum up all the rules and ABIs and sizes for all
> > the
> > >>>> targets I need and generate different IR for each, am I correct?
> > >>>
> > >>> This is a long-known limitation of LLVM IR and there are a lot of
> > >>> proposals to fix it.  It would be great if the LLVM Foundation
> > would
> > >>> fund someone to do the work, as it isn’t a sufficiently high
> > >>> priority for any of the large LLVM consumers and would make a
> huge
> > >>> difference to the utility of LLVM for a lot of people.
> > >> …
> > >>> I think it would be difficult to do it within the timescale of
> the
> > >>> GSoC unless the student was already an experienced LLVM
> developer.
> > >>> It would likely involve designing some good APIs (difficult!),
> > >>> refactoring a bunch of Clang code, and creating a new LLVM
> library.
> > >>> I’ve not seen a GSoC project on this scale succeed in any of the
> > >>> open source projects that I’ve been involved with.  If we had a
> > good
> > >>> design doc and a couple of engaged mentors then it might stand a
> > >>> chance.
> > >>
> > >> Is there a specific design that you think would work?  One of the
> > >> major problems with this sort of proposal is that you need the
> > entire
> > >> clang type system to do this, which means it depends on a huge
> > >> chunk of the Clang AST.  At that point, this isn’t a small library
> > >> that clang uses, this is a library layered on top of Clang itself.
> > >
> > > Given that ABIs are defined in terms of C (and sometimes now C++)
> > language constructs, I think that something like this is the best of
> > all bad options. Really, however, it depends only on the AST and
> > CodeGen, and maybe those (along with 'Basic', etc.) could be made
> into
> > a separately-compilable library. Along with an easy ASTBuilder for C
> > types and function declarations we should be able to satisfy this use
> > case.
> >
> > Indeed.  Today, I can go and get the MIPS, ARM, x86-64, or whatever
> > ABI specification and it defines how all of the C types map to
> > in-memory types and where arguments go.  We currently have no
> standard
> > for how any of this is represented in IR, and I have to look at what
> > clang generates if I want to generate C-compatible IR (and this is
> not
> > stable over time - the contract between clang and the x86 back end
> has
> > changed at least once that I remember).  The minimum that you need to
> > be able to usefully interoperate with C is:
> >
> > - The ability to map each of the C types (int, long, float, double)
> to
> > the corresponding LLVM type.
> >
> > - The ability to generate an LLVM struct that corresponds to a
> > particular C struct (including loads and stores from struct members)
> >
> > - The ability to construct functions that have a C API signature and
> > call functions that have such a signature.
> >
> > We’ve discussed possible APIs for this in the Cambridge LLVM Socials
> a
> > couple of times.  I think that the best proposal was along the
> > following lines:
> >
> > A new CABIBuilder that handles constructing C ABI elements.  This
> > would have the primitive C types as static fields and would allow you
> > to construct a C struct type by passing C types (primitives or other
> > structs, optionally with array sizes).  From this it would construct
> > an LLVM struct and provide IRBuilder-like methods for constructing
> > GEPs to specific fields (and probably loads and stores to bitfields).
> >
> > The same approach would be used for functions and calls.  Once you’ve
> > built the CFunctionType from C structs and primitives for arguments,
> > you would have an analogue of IRBuilder’s CreateCall / CreateInvoke
> > that would take the IR types that correspond to the C types and
> > marshal them correctly.
> >
> > On the other side of the call (constructing a C ABI function by
> > passing a set of C types to the builder), you’d get an LLVM Function
> > that took the arguments in whatever LLVM expects and then stores them
> > into Allocas, which would be returned to the callee, so the front-end
> > author would never need to look at the explicit parameters.
> >
> > You’d need a small subset of Clang’s AST for this (none of the stuff
> > for builtins, nothing for C++ / Objective-C, and so on) and several
> of
> > the bits of CodeGen (in particular, CGTargetInfo contains a bunch of
> > stuff that really should be in LLVM, for example with respect to
> > variadics).  It’s a big bit of refactoring work, and a lot of it
> would
> > probably need to end up duplicated in both clang and LLVM (though it
> > should be easy to automate the testing).
> >
> > Another alternative is to expose these APIs via from Clang itself, so
> > if you need them then you will have to link clang’s Basic, AST and
> > CodeGen libraries (which is only about 10MB in a release build and
> > could be dynamically linked if they’re used by multiple things).
> This
> > approach would also make it easier to extend the interfaces to allow
> > header parsing and C++ interop (which would be nice for using things
> > like shared_ptr across FFI boundaries).
> >
> > David
> >
> 
> I'd prefer not to expose them via Clang, that would just make another
> dependency for those of use generating our own IR directly to LLVM
> without having Clang.  Right now, we have a converter from our own
> backend IR to LLVM IR so we can port our entire compiler suite of
> BASIC, COBOL, Pascal, Fortran, C.  We've had to peek at what Clang
> generates to figure out the correct mapping for us.
> 
> Even with some CABIBuilder, I think we might have to go deeper for us.
> For example, we aren't using IRBuilder at the moment since we end up
> building the IR in pieces and stitching it all back together at the end
> of the conversion process.
> 
> 
> 
> 
> ------------------------------
> 
> Message: 3
> Date: Tue, 1 Nov 2016 14:59:43 +0000
> From: "Robinson, Paul via llvm-dev" <llvm-dev at lists.llvm.org>
> To: Alex Bradbury <asb at asbradbury.org>
> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] RFC: Improving the experience of first-time
> 	contributors
> Message-ID:
> 	<E3B07FDB86BFF041819DC057DEED8FEAF48D64D1 at USCULXMSG08.am.sony.com
> >
> Content-Type: text/plain; charset="utf-8"
> 
> 
> 
> > -----Original Message-----
> > From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> > Alex Bradbury via llvm-dev
> > Sent: Tuesday, November 01, 2016 6:17 AM
> > To: llvm-dev
> > Subject: [llvm-dev] RFC: Improving the experience of first-time
> > contributors
> >
> > Hi all,
> >
> > Some discussions the night before prompted me to do a short lightning
> > talk on 'improving the experience of first-time contributors' at the
> > LLVM Cauldron back in September. I intended to write it up as an RFC,
> > but have only just got round to doing so. I've tried to make this
> > email self-contained, but you may still want to look at the slides or
> > recording of the lightning talk.
> >
> > Slides: https://speakerdeck.com/asb/can-we-improve-the-experience-of-
> > first-time-llvm-contributors
> > Video: https://www.youtube.com/watch?v=32fmbEI9WrM
> >
> > # The problem
> > As a first-time contributor, it's easy to be put off during the patch
> > submission process. e.g. the patch may sit unreviewed for weeks.
> > Typically people new to the project haven't yet reviewed 'review
> > currency', and so their patches are less likely to be picked out by
> > others scanning through llvm-commits. Obviously nobody is doing
> > anything wrong here, it's just likely to be off-putting to newcomers.
> > Even if feedback is negative, it's valuable to know someone has at
> > least looked at your code.
> 
> It's not just first-time contributors who can have this problem...
> ...but it can be especially off-putting to first-timers.
> 
> >
> > # How do other projects solve this?
> > I've always been impressed at how Rust has been able to attract a
> > steady stream of new contributors. They have a simple but seemingly
> > effective tool for helping people land their first patch: the
> highfive
> > bot. Upon a user making their first pull request, it posts a welcome
> > message and assigns someone to review it.
> 
> Sweet.
> 
> >
> > # A proposal
> > I propose that we form a group of people who specifically volunteer
> to
> > help review first-time patches. A simple phabricator bot could assign
> > someone randomly much like Rust's highfive.
> 
> That's an awesome idea for people who do actually post a patch to
> phabricator.  I'll just point out that phabricator isn't officially
> required, and is in fact an extra barrier to entry, so we should have
> some tactic for handling emailed patches if we can manage it.
> 
> I know people will react with "what barrier?" but it's YADA (Yet
> Another Damn Account) and I know *I* hate doing that for something I
> view as a one-off.
> 
> >
> > One potential pitfall I'd like to highlight is that many patches
> > really require the sign-off of the code owner. It could be equally
> > frustrating for newcomers if people helpfully tell them to tweak
> their
> > variable names, fix their formatting, and make other cosmetic changes
> > only to find that time was wasted when the code owner step in a week
> > or two later to say a completely different approach needs to be used.
> 
> People who sign up to be first-responders :-) should probably have some
> sense of when patches are of that nature, and say so in their initial
> response.  If nothing else, cosmetic feedback will get new contributors
> in tune with our style guide and make subsequent rounds feel more
> substantive/less picky.
> 
> >
> > # Next steps
> >
> > Does anybody have any views here? My 'raise your hand' poll at the
> > LLVM Cauldron did seem to think many felt things could be improved.
> 
> Agreed.  Partly it's the raw size of the project; when I was on a two-
> person team it was never a problem to get reviews done, but with a
> dozen or more and looking for someone to volunteer, there's a
> significant "somebody else will get to it" factor.  And then nobody
> does.  With volunteer first-responders it's something they've signed up
> to do, however, and that makes a big difference.
> --paulr
> 
> > I'll be at the LLVM Dev Meeting this week and would love to discuss
> > further. Perhaps the BoF on raising the next generation of LLVM
> > developers would be an appropriate venue?
> > http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
> >
> > Best,
> >
> > Alex
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> ------------------------------
> 
> Message: 4
> Date: Tue, 01 Nov 2016 09:28:48 -0700
> From: Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org>
> To: Hal Finkel <hfinkel at anl.gov>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>,
> 	ジョウェットジェームス  <b3i4zz1gu1 at docomo.ne.jp>
> Subject: Re: [llvm-dev] What was the IR made for precisely?
> Message-ID: <CC5D15F2-0A78-4A58-9A94-0AF7089DD7C3 at apple.com>
> Content-Type: text/plain; charset="utf-8"
> 
> 
> > On Oct 31, 2016, at 10:53 AM, Hal Finkel via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> >
> > ----- Original Message -----
> >> From: "Reid Kleckner" <rnk at google.com <mailto:rnk at google.com>>
> >> To: "Hal Finkel" <hfinkel at anl.gov <mailto:hfinkel at anl.gov>>
> >> Cc: "Chris Lattner" <clattner at apple.com
> <mailto:clattner at apple.com>>, "llvm-dev" <llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>>, "ジョウェットジェームス"
> >> <b3i4zz1gu1 at docomo.ne.jp <mailto:b3i4zz1gu1 at docomo.ne.jp>>
> >> Sent: Monday, October 31, 2016 11:39:04 AM
> >> Subject: Re: [llvm-dev] What was the IR made for precisely?
> >>
> >>
> >> I think it would be possible to build a stripped-down
> >> C-with-extensions-only type representation that represents only the
> >> types that have interesting ABI implications. This includes several
> >> things not present in LLVM IR types: _Complex, union, flexible array
> >> members, alignment attributes, etc. We could either then use that
> >> library to generate LLVM IR, or embed it in the IR for use at call
> >> sites.
> >>
> >
> > I think this is certainly possible. The question is: Is Clang going
> to use this representation? If not, who will want to maintain this
> separately from Clang?
> >
> > To a large extent, this seems like factoring out the logic in Clang's
> lib/CodeGen/TargetInfo* (as well as lib/Basic/Target* and a few other
> places) into a reusable form.
> 
> I share this view, and I’d like a solution that would be used by clang
> by default! (Mostly for the reason hinted above in the question: “who
> will want to maintain this separately from Clang?”)
> 
> —
> Mehdi
> 
> >
> >>
> >> You can exclude C++ because, for the most part, C++ types are either
> >> passed indirectly or as the analogous C struct would be passed. Some
> >> work would be needed to support packing fields into base tail
> >> padding, but that's not too bad.
> >>
> >>
> >> On Fri, Oct 28, 2016 at 1:25 PM, Hal Finkel via llvm-dev <
> >> llvm-dev at lists.llvm.org > wrote:
> >>
> >>
> >>
> >>
> >> ----- Original Message -----
> >>> From: "Chris Lattner via llvm-dev" < llvm-dev at lists.llvm.org >
> >>> To: "David Chisnall" < David.Chisnall at cl.cam.ac.uk >
> >>> Cc: llvm-dev at lists.llvm.org , "ジョウェットジェームス" <
> >>> b3i4zz1gu1 at docomo.ne.jp >
> >>> Sent: Friday, October 28, 2016 2:13:06 PM
> >>> Subject: Re: [llvm-dev] What was the IR made for precisely?
> >>>
> >>>
> >>>> On Oct 28, 2016, at 1:21 AM, David Chisnall <
> >>>> David.Chisnall at cl.cam.ac.uk > wrote:
> >>>>
> >>>> On 28 Oct 2016, at 02:43, ジョウェットジェームス <
> b3i4zz1gu1 at docomo.ne.jp >
> >>>> wrote:
> >>>>>
> >>>>> I would need to sum up all the rules and ABIs and sizes for all
> >>>>> the targets I need and generate different IR for each, am I
> >>>>> correct?
> >>>>
> >>>> This is a long-known limitation of LLVM IR and there are a lot of
> >>>> proposals to fix it. It would be great if the LLVM Foundation
> would
> >>>> fund someone to do the work, as it isn’t a sufficiently high
> >>>> priority for any of the large LLVM consumers and would make a huge
> >>>> difference to the utility of LLVM for a lot of people.
> >>> …
> >>>> I think it would be difficult to do it within the timescale of the
> >>>> GSoC unless the student was already an experienced LLVM developer.
> >>>> It would likely involve designing some good APIs (difficult!),
> >>>> refactoring a bunch of Clang code, and creating a new LLVM
> library.
> >>>> I’ve not seen a GSoC project on this scale succeed in any of the
> >>>> open source projects that I’ve been involved with. If we had a
> good
> >>>> design doc and a couple of engaged mentors then it might stand a
> >>>> chance.
> >>>
> >>> Is there a specific design that you think would work? One of the
> >>> major problems with this sort of proposal is that you need the
> >>> entire clang type system to do this, which means it depends on a
> >>> huge chunk of the Clang AST. At that point, this isn’t a small
> >>> library that clang uses, this is a library layered on top of Clang
> >>> itself.
> >>
> >> Given that ABIs are defined in terms of C (and sometimes now C++)
> >> language constructs, I think that something like this is the best of
> >> all bad options. Really, however, it depends only on the AST and
> >> CodeGen, and maybe those (along with 'Basic', etc.) could be made
> >> into a separately-compilable library. Along with an easy ASTBuilder
> >> for C types and function declarations we should be able to satisfy
> >> this use case.
> >>
> >> -Hal
> >>
> >>>
> >>> -Chris
> >>> _______________________________________________
> >>> LLVM Developers mailing list
> >>> llvm-dev at lists.llvm.org
> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>
> >>
> >> --
> >> Hal Finkel
> >> Lead, Compiler Technology and Programming Languages Leadership
> >> Computing Facility Argonne National Laboratory
> >>
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >>
> >
> > --
> > Hal Finkel
> > Lead, Compiler Technology and Programming Languages Leadership
> > Computing Facility Argonne National Laboratory
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.llvm.org/pipermail/llvm-
> dev/attachments/20161101/778da195/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 5
> Date: Tue, 1 Nov 2016 09:33:08 -0700
> From: Dehao Chen via llvm-dev <llvm-dev at lists.llvm.org>
> To: "Robinson, Paul" <paul.robinson at sony.com>
> Cc: "llvm-dev at lists.llvm.org" <llvm-dev at lists.llvm.org>, Xinliang
> 	David Li <davidxl at google.com>
> Subject: Re: [llvm-dev] (RFC) Encoding code duplication factor in
> 	discriminator
> Message-ID:
> 	<CAO2gOZUptwfpqfXKFEssypmRt39qiQP+O+6qMmRUo4nkZYHQig at mail.gmail.c
> om>
> Content-Type: text/plain; charset="utf-8"
> 
> On Thu, Oct 27, 2016 at 2:53 PM, Robinson, Paul
> <paul.robinson at sony.com>
> wrote:
> 
> > It looks like the example doesn't use the encoding described in the
> text?
> >
> >
> >
> > Assume that the discriminator is uint32. The traditional
> discriminator
> > is less than 256, let's take 8 bit for it. For duplication factor
> > (type 1 duplication), we assume the maximum unroll_factor *
> > vectorize_factor is less than 256, thus 8 bit for it. For unique
> > number(type 2 duplication), we assume code is at most duplicated 32
> > times, thus 5 bit for it. Overall, we still have 11 free bits left in
> the discriminator encoding.
> >
> >
> >
> > Let's take the original source as an example, after loop unrolling
> and
> > peeling, the code may looks like:
> >
> >
> >
> > for (i = 0; i < N & 3; i+= 4) {
> >
> >   foo();  // discriminator: 0x40
> >
> >   foo();  // discriminator: 0x40
> >
> >   foo();  // discriminator: 0x40
> >
> >   foo();  // discriminator: 0x40
> >
> > }
> >
> > if (i++ < N) {
> >
> >   foo();   // discriminator: 0x100
> >
> >   if (i++ < N) {
> >
> >     foo(); // discriminator: 0x200
> >
> >     if (i++ < N) {
> >
> >       foo();  // discriminator: 0x300
> >
> >     }
> >
> >   }
> >
> > }
> >
> >
> >
> > If we allocate 8 bits to "traditional" discriminators, then 0x40
> falls
> > into that range, so I'd think the calls to foo() inside the loop
> > should be using 0x400 to encode the unroll factor.
> >
> 
> You are right, thanks for pointing this out.
> 
> 
> > Note this requires 2 bytes for ULEB128 instead of 1.
> >
> > And if we allocate another 8 bits to the unroll factor, then the
> > trailing calls should use 0x10000, 0x20000, 0x30000.  These will
> > require 3 bytes for
> > ULEB128 instead of 2.
> >
> >
> >
> > I think it would be fine to allocate only 4 bits to "traditional"
> > discriminators (as you need them to be unique across the same source
> > location, but not across different source locations, and 16
> > independent basic blocks for the same source location seems like
> > plenty to me; but I haven't looked at a lot of cases with
> > discriminators).  This would allow you to use 0x40 to encode the
> > unroll factor in this example.  If you still want to allocate 8 bits
> > for unroll factor, then the trailing calls would use 0x1000, 0x2000,
> > 0x3000 so as long as you had no more than 3 trailing calls you can
> still encode the discriminator in a 2-byte ULEB128.
> >
> 
> I did some evaluation on speccpu2006 c/c++ benchmarks. Among all source
> built, there are 268K non-zero discriminators emitted. The largest
> discriminator is 779 (coming from 471.omnetpp, which has significant
> amount of EH code.) The 99% percentile is 18. The 85% percentile is 3.
> For the duplication factor, the largest is 320, the 99% percentile is
> 40, the 85% percentile is 4.
> 
> If we want to encode this into the discriminator, I would propose
> encode something like:
> 
> high bits   ---------->  low bits
> EEEEEEEECCCCCFFDDD CCCFFFFFFDDDDD
> 
> D: normal discriminator
> F: duplication factor
> C: code cloning
> E: empty bits
> 
> So the lower 14 bits should be able to cover 99% percentile
> 
> Or something like:
> high bits   ---------->  low bits
> EEEEEEEECCCCCFFDDD CFFFDDD CCFFFDD
> 
> So the lower 7 bits should be able to cover 85% percentile and the
> lower 14 bits should be able to cover 99% percentile.
> 
> Dehao
> 
> 
> 
> > --paulr
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.llvm.org/pipermail/llvm-
> dev/attachments/20161101/151d1ad3/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 6
> Date: Wed, 2 Nov 2016 00:33:19 +0800
> From: 雷景然 via llvm-dev <llvm-dev at lists.llvm.org>
> To: "=?gb18030?B?bGx2bS1kZXY=?=" <llvm-dev at lists.llvm.org>
> Subject: [llvm-dev] HowToRunJIT doesn't work on windows 10 x64
> Message-ID: <tencent_60F8DC4C3E345C5C46654C39 at qq.com>
> Content-Type: text/plain; charset="gb18030"
> 
> Hi All,
> 
> 
> First, I want to show you a bug which I found at HowToUseJIT.cpp.
> 
> 
> This is correct one, I changed locally:
> ----------------------------
>   outs() << "We just constructed this LLVM module:\n\n" << *M;
>   outs() << "\n\nRunning foo on : " << M->getTargetTriple();
>   outs().flush();
> 
> 
>   // Now we create the JIT.
>   ExecutionEngine* EE = EngineBuilder(std::move(Owner)).create();
> 
> ----------------------------
> 
> 
> And then, the original version is :
> ----------------------------
>   // Now we create the JIT.
>   ExecutionEngine* EE = EngineBuilder(std::move(Owner)).create();
> 
> 
>   outs() << "We just constructed this LLVM module:\n\n" << *M;
>   outs() << "\n\nRunning foo on : " << M->getTargetTriple();
>   outs().flush();
> 
> 
> ----------------------------
> 
> 
> Because `Owner` is move into `EE`, and `M` is came from `Owner`, so the
> original version is incorrect.
> 
> 
> I had change this locally. :)
> 
> 
> Finally, My question is : why I got the error below when I ran the
> HowToUseJIT
> -----------------------------------------
> We just constructed this LLVM module:
> 
> 
> ; ModuleID = 'test'
> source_filename = "test"
> target triple = "x86_64-pc-windows-msvc"
> 
> 
> define i32 @add1(i32 %AnArg) {
> EntryBlock:
>   %0 = add i32 1, %AnArg
>   ret i32 %0
> }
> 
> 
> define i32 @foo() {
> EntryBlock:
>   %0 = tail call i32 @add1(i32 10)
>   ret i32 %0
> }
> 
> 
> 
> 
> 
> 
> Running foo on : x86_64-pc-windows-msvc
> LLVM ERROR: Target does not support MC emission!
> Press any key to continue . . .
> 
> -----------------------------------------
> 
> 
> My HowToUseJIT.cpp code is below:
> --------------------------
> int main() {
>   ...
>   M->setTargetTriple(sys::getDefaultTargetTriple());
>   outs() << "We just constructed this LLVM module:\n\n" << *M;
>   outs() << "\n\nRunning foo on : " << M->getTargetTriple();
>   outs().flush();
> 
> 
>   ForceMCJITLinking::ForceMCJITLinking();
>   // Now we create the JIT.
>   ExecutionEngine* EE = EngineBuilder(std::move(Owner)).create();
> 
> 
>   // Call the `foo' function with no arguments:
>   std::vector<GenericValue> noargs;
>   GenericValue gv = EE->runFunction(FooF, noargs);
> 
>   ...
> }
> --------------------------
> 
> 
> 
> 
> Best Regards.
> Leagmain  (Jingran.Lear)
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.llvm.org/pipermail/llvm-
> dev/attachments/20161102/01cd5d28/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 7
> Date: Tue, 01 Nov 2016 09:42:17 -0700
> From: Mehdi Amini via llvm-dev <llvm-dev at lists.llvm.org>
> To: Alex Bradbury <asb at asbradbury.org>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] RFC: Improving the experience of first-time
> 	contributors
> Message-ID: <407325AB-6A6E-4604-A9DC-3126DA4E890D at apple.com>
> Content-Type: text/plain; charset="utf-8"
> 
> 
> > On Nov 1, 2016, at 6:17 AM, Alex Bradbury via llvm-dev <llvm-
> dev at lists.llvm.org> wrote:
> >
> > Hi all,
> >
> > Some discussions the night before prompted me to do a short lightning
> > talk on 'improving the experience of first-time contributors' at the
> > LLVM Cauldron back in September. I intended to write it up as an RFC,
> > but have only just got round to doing so. I've tried to make this
> > email self-contained, but you may still want to look at the slides or
> > recording of the lightning talk.
> >
> > Slides:
> > https://speakerdeck.com/asb/can-we-improve-the-experience-of-first-
> tim
> > e-llvm-contributors
> > Video: https://www.youtube.com/watch?v=32fmbEI9WrM
> > <https://www.youtube.com/watch?v=32fmbEI9WrM>
> 
> Great initiative! Thanks.
> 
> Have you seen this past discussion:
> http://lists.llvm.org/pipermail/llvm-dev/2016-March/097027.html
> I think the main idea has been to set-up a regular IRC session
> dedicated to “new contributors”.
> 
> >
> > # The problem
> > As a first-time contributor, it's easy to be put off during the patch
> > submission process. e.g. the patch may sit unreviewed for weeks.
> > Typically people new to the project haven't yet reviewed 'review
> > currency', and so their patches are less likely to be picked out by
> > others scanning through llvm-commits. Obviously nobody is doing
> > anything wrong here, it's just likely to be off-putting to newcomers.
> > Even if feedback is negative, it's valuable to know someone has at
> > least looked at your code.
> >
> > # How do other projects solve this?
> > I've always been impressed at how Rust has been able to attract a
> > steady stream of new contributors. They have a simple but seemingly
> > effective tool for helping people land their first patch: the
> highfive
> > bot. Upon a user making their first pull request, it posts a welcome
> > message and assigns someone to review it.
> >
> > # A proposal
> > I propose that we form a group of people who specifically volunteer
> to
> > help review first-time patches. A simple phabricator bot could assign
> > someone randomly much like Rust's highfive.
> >
> > One potential pitfall I'd like to highlight is that many patches
> > really require the sign-off of the code owner.
> 
> I don’t believe it is exact, unless you used “code owner” in a broader
> sense that what we mean usually with CODE_OWNERS.txt ; as you can see
> here: http://llvm.org/docs/DeveloperPolicy.html#code-owners "The sole
> responsibility of a code owner is to ensure that a commit to their area
> of the code is appropriately reviewed, either by themself or by someone
> else."
> You may need a sign-off from a “core contributor” well aware of the
> particular area you’re touching. And even that depends on the level of
> the changes: a small bug fix is less involved than something that
> requires large change or involves some design.
> 
> 
> > It could be equally
> > frustrating for newcomers if people helpfully tell them to tweak
> their
> > variable names, fix their formatting, and make other cosmetic changes
> > only to find that time was wasted when the code owner step in a week
> > or two later to say a completely different approach needs to be used.
> 
> I hope the “group of volunteers” will be well aware of LLVM and be able
> to spot when they need to pull-in someone else for a particular review
> (e.g. an important change in the new pass manager is likely to require
> Chandler to sign-off).
> 
> Best,
> 
> —
> Mehdi
> 
> 
> 
> >
> > # Next steps
> >
> > Does anybody have any views here? My 'raise your hand' poll at the
> > LLVM Cauldron did seem to think many felt things could be improved.
> > I'll be at the LLVM Dev Meeting this week and would love to discuss
> > further. Perhaps the BoF on raising the next generation of LLVM
> > developers would be an appropriate venue?
> > http://lists.llvm.org/pipermail/llvm-dev/2016-October/106658.html
> >
> > Best,
> >
> > Alex
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.llvm.org/pipermail/llvm-
> dev/attachments/20161101/62717283/attachment-0001.html>
> 
> ------------------------------
> 
> Message: 8
> Date: Tue, 1 Nov 2016 09:43:41 -0700
> From: Dehao Chen via llvm-dev <llvm-dev at lists.llvm.org>
> To: Hal Finkel <hfinkel at anl.gov>
> Cc: llvm-dev <llvm-dev at lists.llvm.org>, Xinliang David Li
> 	<davidxl at google.com>
> Subject: Re: [llvm-dev] (RFC) Encoding code duplication factor in
> 	discriminator
> Message-ID:
> 	<CAO2gOZWbFVa-
> ZopougMbSx9oTRsMuK4nxzrw0t7W0AAeQ3cmrw at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
> 
> On Fri, Oct 28, 2016 at 3:07 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
> > Hi Dehao,
> >
> > This is definitely an important problem, thanks for writing this up!
> >
> > There is a related problem that I think we can address at the same
> time:
> > When we multiversion code, for example when we use runtime checks to
> enable
> > the creation of a vectorized loop while retaining the scalar loop,
> and then
> > we collect profiling data, we should be able to recover the relative
> > running time of the different versions of the loop. In the future, I
> > suspect we'll end up with transformations that produce even more
> versions
> > of loops (Intel's compiler, for example, has a useful pragma that
> allows
> > the user to request loop specializations for a specified set of trip
> > counts).
> >
> > I'd like to have a scheme where the source location + discriminator
> can be
> > mapped to information about the relevant loop version so that our
> profiling
> > tools can display that information usefully to the user, and so that
> our
> > optimizations can make useful choices (i.e. don't bother vectorizing
> a loop
> > when the scalar loop is run a lot but the vectorized version almost
> never
> > runs).
> >
> 
> That's definitely a valid and important use case, and it is important
> to
> sample pgo too. That's why I proposed to have "duplicated code that may
> have different execution count" being recorded. Will that suffice to
> get
> the info you want? (i.e. for every version of the multi-versioned loop,
> you
> will have a disincentive discriminator associated with all the code it
> expands.
> 
> 
> >
> > In short, I think that differentiating these different regions using
> the
> > descriminator seems like a natural choice, but "And we do not need to
> > introduce new building blocks to debug info" might not save us all
> that
> > much in the long run. To keep information on what regions correspond
> to
> > what optimizations, we may need to do that. That's not a bad thing,
> and I'd
> > rather we solve this in a way that is extensible. Plus, this might
> make it
> > easier to use fewer bits, thus helping the overall impact on the size
> of
> > the debug sections.
> >
> 
> I agree that if we want to extend this in the future, we need to have
> separate dwarf bits other than discriminator. For current use case,
> discriminator seem to be good enough. And if we encode efficiently, it
> will
> be better than introducing a new field. e.g., we can encode all info in
> a
> 1-byte ULEB128 85%~90% of the time, but for a new field, we will at
> least
> need 2 bytes if both discriminator and cloning info exists for an
> instruction.
> 
> Dehao
> 
> 
> >
> > Thanks again,
> > Hal
> >
> > ----- Original Message -----
> > > From: "Dehao Chen via llvm-dev" <llvm-dev at lists.llvm.org>
> > > To: llvm-dev at lists.llvm.org
> > > Cc: "Xinliang David Li" <davidxl at google.com>
> > > Sent: Thursday, October 27, 2016 1:39:15 PM
> > > Subject: [llvm-dev] (RFC) Encoding code duplication factor in
> > discriminator
> > >
> > > Motivation:
> > > Many optimizations duplicate code. E.g. loop unroller duplicates
> the
> > > loop body, GVN duplicates computation, etc. The duplicated code
> will
> > > share the same debug info with the original code. For SamplePGO,
> the
> > > debug info is used to present the profile. Code duplication will
> > > affect profile accuracy. Taking loop unrolling for example:
> > >
> > >
> > > #1 foo();
> > > #2 for (i = 0; i < N; i++) {
> > >
> > > #3 bar();
> > > #4 }
> > >
> > >
> > > If N is 8 during runtime, a reasonable profile will look like:
> > >
> > >
> > > #1: 10
> > > #3: 80
> > >
> > >
> > >
> > > But if the compiler unrolls the loop by a factor of 4, the callsite
> > > to bar() is duplicated 4 times and the profile will look like:
> > >
> > >
> > > #1: 10
> > > #3: 20
> > >
> > >
> > > The sample count for #3 is 20 because all 4 callsites to bar() are
> > > sampled 20 times each, and they shared the same debug loc (#3) so
> > > that 20 will be attributed to #3 (If one debugloc is mapped by
> > > multiple instructions, the max sample count of these instructions
> is
> > > used as debugloc's sample count).
> > >
> > >
> > > When loading this profile into compiler, it will think the loop
> trip
> > > count is 2 instead of 8.
> > >
> > >
> > > Proposal:
> > > When compiler duplicates code, it encodes the duplication info in
> the
> > > debug info. As the duplication is not interesting to debugger, I
> > > propose to encode this as part of the discriminator.
> > >
> > >
> > > There are 2 types of code duplication:
> > >
> > >
> > > 1. duplicated code are guaranteed to have the same execution count
> > > (e.g. loop unroll and loop vectorize). We can record the
> duplication
> > > factor, for the above example "4" is recorded in the discriminator.
> > > 2. duplicated code that may have different execution count (e.g.
> loop
> > > peel and gvn). For a same debugloc, a unique number is assigned to
> > > each copy and encoded in the discriminator.
> > >
> > >
> > > Assume that the discriminator is uint32. The traditional
> > > discriminator is less than 256, let's take 8 bit for it. For
> > > duplication factor (type 1 duplication), we assume the maximum
> > > unroll_factor * vectorize_factor is less than 256, thus 8 bit for
> > > it. For unique number(type 2 duplication), we assume code is at
> most
> > > duplicated 32 times, thus 5 bit for it. Overall, we still have 11
> > > free bits left in the discriminator encoding.
> > >
> > >
> > > Let's take the original source as an example, after loop unrolling
> > > and peeling, the code may looks like:
> > >
> > >
> > > for (i = 0; i < N & 3; i+= 4) {
> > > foo(); // discriminator: 0x40
> > > foo(); // discriminator: 0x40
> > > foo(); // discriminator: 0x40
> > > foo(); // discriminator: 0x40
> > > }
> > > if (i++ < N) {
> > > foo(); // discriminator: 0x100
> > > if (i++ < N) {
> > > foo(); // discriminator: 0x200
> > > if (i++ < N) {
> > > foo(); // discriminator: 0x300
> > > }
> > > }
> > > }
> > >
> > >
> > > The cost of this change would be increased debug_line size because:
> > > 1. we are recording more discriminators 2. discriminators become
> > > larger and will take more ULEB128 encoding.
> > >
> > >
> > > The benefit is that the sample pgo profile can accurately represent
> > > the code execution frequency. And we do not need to introduce new
> > > building blocks to debug info.
> > >
> > >
> > > Comments?
> > >
> > >
> > > Thanks,
> > > Dehao
> > > _______________________________________________
> > > LLVM Developers mailing list
> > > llvm-dev at lists.llvm.org
> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> > >
> >
> > --
> > Hal Finkel
> > Lead, Compiler Technology and Programming Languages
> > Leadership Computing Facility
> > Argonne National Laboratory
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <http://lists.llvm.org/pipermail/llvm-
> dev/attachments/20161101/78e6d8b9/attachment.html>
> 
> ------------------------------
> 
> Subject: Digest Footer
> 
> _______________________________________________
> llvm-dev mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 
> 
> ------------------------------
> 
> End of llvm-dev Digest, Vol 149, Issue 2
> ****************************************