[llvm-dev] [Openmp-dev] [cfe-dev] RFC: Proposing an LLVM subproject for parallelism runtime and support libraries

Wed Jun 1 14:45:22 PDT 2016

----- Original Message -----
> From: "C Bergström" <cbergstrom at pathscale.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev" <cfe-dev at lists.llvm.org>, "openmp-dev"
> <openmp-dev at lists.llvm.org>, "Chandler Carruth" <chandlerc at gmail.com>, "Carlo Bertolli" <cbertol at us.ibm.com>,
> "Andrey Bokhanko" <andreybokhanko at gmail.com>
> Sent: Wednesday, June 1, 2016 10:53:27 AM
> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an LLVM subproject for parallelism runtime and support
> libraries
> 
> On Wed, Jun 1, 2016 at 11:38 PM, Hal Finkel <hfinkel at anl.gov> wrote:
> > ----- Original Message -----
> >> From: "C Bergström" <cbergstrom at pathscale.com>
> >> To: "Hal Finkel" <hfinkel at anl.gov>
> >> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev"
> >> <cfe-dev at lists.llvm.org>, "openmp-dev"
> >> <openmp-dev at lists.llvm.org>, "Chandler Carruth"
> >> <chandlerc at gmail.com>, "Carlo Bertolli" <cbertol at us.ibm.com>,
> >> "Andrey Bokhanko" <andreybokhanko at gmail.com>
> >> Sent: Wednesday, June 1, 2016 9:42:34 AM
> >> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing an
> >> LLVM subproject for parallelism runtime and support
> >> libraries
> >>
> >> On Wed, Jun 1, 2016 at 8:52 PM, Hal Finkel <hfinkel at anl.gov>
> >> wrote:
> >> > ----- Original Message -----
> >> >> From: "C Bergström" <cbergstrom at pathscale.com>
> >> >> To: "Hal Finkel" <hfinkel at anl.gov>
> >> >> Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev"
> >> >> <cfe-dev at lists.llvm.org>, "openmp-dev"
> >> >> <openmp-dev at lists.llvm.org>, "Chandler Carruth"
> >> >> <chandlerc at gmail.com>, "Carlo Bertolli" <cbertol at us.ibm.com>,
> >> >> "Andrey Bokhanko" <andreybokhanko at gmail.com>
> >> >> Sent: Wednesday, June 1, 2016 6:46:57 AM
> >> >> Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing
> >> >> an
> >> >> LLVM subproject for parallelism runtime and support
> >> >> libraries
> >> >>
> >> >> On Wed, Jun 1, 2016 at 7:22 PM, Hal Finkel <hfinkel at anl.gov>
> >> >> wrote:
> >> >> >
> >> >> > ________________________________
> >> >> >
> >> >> > From: "C Bergström" <cbergstrom at pathscale.com>
> >> >> > To: "Hal Finkel" <hfinkel at anl.gov>
> >> >> > Cc: "llvm-dev" <llvm-dev at lists.llvm.org>, "cfe-dev"
> >> >> > <cfe-dev at lists.llvm.org>, "openmp-dev"
> >> >> > <openmp-dev at lists.llvm.org>,
> >> >> > "Chandler Carruth" <chandlerc at gmail.com>, "Carlo Bertolli"
> >> >> > <cbertol at us.ibm.com>, "Andrey Bokhanko"
> >> >> > <andreybokhanko at gmail.com>
> >> >> > Sent: Wednesday, June 1, 2016 12:19:19 AM
> >> >> > Subject: Re: [Openmp-dev] [llvm-dev] [cfe-dev] RFC: Proposing
> >> >> > an
> >> >> > LLVM
> >> >> > subproject for parallelism runtime and support libraries
> >> >> >
> >> >> > The thread has lost focus and cherry picking replies..
> >> >> >
> >> >> > To restate things since maybe you missed my points
> >> >> > ----
> >> >> > 1) SE is a programming model and needs a home of it's own.
> >> >> > Having a
> >> >> > programming model with it's headers and all other stuff glued
> >> >> > into
> >> >> > a runtime
> >> >> > project which intends to be universal and PM agnostic doesn't
> >> >> > make
> >> >> > sense.
> >> >> >
> >> >> > They'd start in separate subdirectories.
> >> >> >
> >> >> >
> >> >> > 1.1) The more I look, the most it seems SE is just a
> >> >> > step-child
> >> >> > project and
> >> >> > stuffing it in llvm while still not having any users or
> >> >> > strong
> >> >> > backing
> >> >> > doesn't make sense. We have enough PM already and my gut
> >> >> > feeling
> >> >> > is
> >> >> > this
> >> >> > isn't going in a direction to bring in other stakeholders.
> >> >>
> >> >> Point by point because I don't agree with what you write below
> >> >> entirely
> >> >>
> >> >> >
> >> >> > I think this is the core of my reply. OpenMP has a strong
> >> >> > user
> >> >> > community,
> >> >>
> >> >> I'd agree here
> >> >>
> >> >> > but OpenMP 4 offloading is still young.
> >> >>
> >> >> ditto
> >> >>
> >> >> > OpenMP 4 offloading does not yet
> >> >> > have a real user community
> >> >>
> >> >> agreement
> >> >>
> >> >> > yet because the first implementations just
> >> >> > started shipping very recently.
> >> >>
> >> >> Partially strongly disagree - Maybe you meant on Power8?
> >> >> Intel has had their support for OMP4 for quite some time. It
> >> >> may
> >> >> have
> >> >> been buggy and focused on simd directive, but as best I can
> >> >> tell
> >> >> they
> >> >> have worked quite hard to fix bugs and improve it.
> >> >
> >> > I meant across many platforms. Intel had preliminary support for
> >> > OpenMP 4 offloading directives even before OpenMP 4 was
> >> > standardized (in 2013). This did not include all of what ended
> >> > up
> >> > in the standard, and the offloading part of the standard needed
> >> > to
> >> > be fixed in a breaking way for OpenMP 4.1/4.5 regardless, so
> >> > this
> >> > is certainly the exception rather than the rule.
> >> >
> >> >>
> >> >> (Shameless self promotion) We have had some degree of OMP4
> >> >> offloading
> >> >> for like 1.5 years now.. It's mostly targeting the GPU and x86,
> >> >> but
> >> >> also more recently working on Power8/AArch64.
> >> >>
> >> >> I'm quite certain these companies all have varying degrees of
> >> >> OMP4
> >> >> done: Cray, IBM, PGI
> >> >
> >> > Yes, many are working on implementations, and some have shipped.
> >> > There's a list here: http://openmp.org/wp/openmp-compilers/ - no
> >> > one here really lists any OpenMP 4 offloading implementations
> >> > before 2015. PGI does not currently list OpenMP 4 at all
> >> > (although
> >> > they've certainly done a lot of work on OpenACC).
> >> >
> >> >>
> >> >> > Furthermore, our implementation is certainly
> >> >> > quite new, and OpenMP 4 offloading is really quite akin to SE
> >> >> > in
> >> >> > that
> >> >> > regard.
> >> >>
> >> >> Strongly disagree - Intel has been working with the LLVM
> >> >> community
> >> >> on
> >> >> the parsing/sema and other parts of OMP4 for like a year or
> >> >> more..
> >> >> This has been a long and well vetted process backed back a well
> >> >> defined standard.
> >> >
> >> > Yes, both Intel and IBM have contributed significantly to Clang
> >> > in
> >> > this regard.
> >> >
> >> >> Compared to SE which is just some thing that popped
> >> >> up out of nowhere and
> >> >
> >> > It popped up in TensorFlow, which itself popped up out of
> >> > nowhere,
> >> > but is already a popular open-source project for machine
> >> > learning.
> >> >
> >> >> has a couple people from Google throwing their
> >> >> weight around.
> >> >
> >> > Google is a trusted member of the LLVM community and a major
> >> > contributor. So I agree with you in the following sense: When
> >> > Google promises to update the code to follow LLVM's coding
> >> > conventions and to manage its future development in accordance
> >> > with our community norms, I believe them. I think they've earned
> >> > a
> >> > place in the "trust but verify" category in this regard, and I
> >> > think this is a positive thing.
> >> >
> >>
> >> yes, but not every project that is started is finished and who
> >> would
> >> be the maintainer if the person leaves Google?
> >
> > Good question, although I'm under the impression that there are
> > multiple developers.
> >
> >> I still don't see why
> >> they can't fork it on github or create a project to let it bake
> >> and
> >> get some users or traction.
> >
> > I think they could, but... SE has users, even if they're just
> > inside of Google currently, and some of the relevant code is in a
> > newly-popular open-source project (TensorFlow), and so I infer a
> > certain amount of experience in SE's development team. Will my
> > users want to use SE? I have no idea. I do care about OpenMP's
> > offloading model because I'm certain I'll have users who care
> > deeply about it within a couple of years. So regarding my
> > optimistic attitude you describe below, I'm interested here is
> > getting another set of eyes on the OpenMP offloading
> > implementation, some sharing of expertise, voicing of alternate
> > viewpoints, etc. -- hopefully, in the end, this will benefit both
> > projects. That might influence, for the better, future versions of
> > OpenMP. It might also end up providing a model that better serves
> > (perhaps though another abstraction) my C++-based users for which
> > OpenMP currently does not work well. None of this might happen,
> > but it definitely won't happen unless we push for it. I don't
> > think we get any of these other benefits it they just stick the
> > code on github and develop it on their own. If they're willing to
> > do this as part of the LLVM community, I think we should take
> > advantage of that.
> >
> >> >> > I view them both as experimental projects, and both have
> >> >> > strong
> >> >> > backing with significant investment, so I expect both to
> >> >> > mature
> >> >> > over time.
> >> >>
> >> >> I'd agree both are experimental and probably need some work.
> >> >> I'd
> >> >> strongly disagree the amount of investment of both projects is
> >> >> equal.
> >> >> Again - OMP4 may have zero real world users, but there's a lot
> >> >> of
> >> >> stubborn people trying to make it work. Compared to SE which
> >> >> the
> >> >> people involved have yet to even answer basic things - like
> >> >> what's
> >> >> the
> >> >> future of SE and how to contribute to it.
> >> >
> >> > They might not know what the future is. They have code that is
> >> > useful to them, and they feel would be useful to others. There's
> >> > strong potential for compiler integration in the future, but it
> >> > is
> >> > just a library now. They've talked about extending it to cover
> >> > different backends (OpenCL, etc.), and investigating
> >> > host-based-task dependencies, etc. My impression is that, in
> >> > general, they'd like the code's users, present and future, to
> >> > shape its future. All of this seems perfectly healthy. If it
> >> > becomes part of LLVM, how to contribute to it will become clear.
> >>
> >> I love your optimistic attitude, but the reality is that
> >> successful
> >> programming models take a significant amount of work. I'm well
> >> aware
> >> that they can handle at least part of that, but so far I'm not
> >> sure
> >> who else will invest in it..
> >>
> >> The industry has : CUDA, OpenCL, OpenMP4, OpenACC, C++AMP,
> >> Parallel
> >> STL(??), some research projects from DOE and dead stuff like
> >> HMPP...
> >> Are all these so broken that they can't be fixed to fill the same
> >> necessity as SE?
> >
> > You're right. SE is certainly different from all of these,
> > higher-level than some and lower-level than others. And, of
> > course, I'm reminded of this: https://xkcd.com/927/
> >
> >>
> >> How similar is it to the failed AMD Stream Computing (sorry AMD)
> >> http://www.ele.uri.edu/courses/ele408/StreamGPU.pdf
> >>
> >> Intel's work had to live and grow outside for quite a while -
> >> Google
> >> may have more good karma,
> >
> > What they have is a potentially-valuable contribution to our
> > project. For me, this is about the strategic health of parts of
> > the project I care about. I have reasons (that I have not been
> > subtle about), that I think there are potential benefits to having
> > both SE and libompoffload together in some sense.
> >
> > To be clear, I am an optimist. I think that different groups can
> > collaborate, or at least share technical discussions, and mutually
> > improve their work. It does not always happen, but we can
> > encourage it by creating an environment in which it is more likely
> > to occur.
> 
> Sorry, but the interaction so far doesn't leave me warm and fuzzy.. 

Clearly ;)

>I
> see them wanting to push their code as-is instead of working more
> closely with OMP and Intel.
> 
> Nothing stops them from reviewing the OpenMP implementation stuff
> today. Since you're friends with them - maybe ask them to help out
> there.

To be clear, I don't currently know the SE developers personally.

> If they had a strong interest in this area, I suspect that
> would have already happened.

I agree. I don't think they necessarily have a strong interest. I think the question here is what the communities's strategic interests are. SE's developers came to us with a proposal, and we can certainly make our expectations clear. That might include help reviewing the OpenMP offloading-library implementation.

> From what you're saying above.. it seems like SE won't succeed unless
> it becomes a project? If that's the case then maybe it should die...
> You have to be _____ this tall to go on the ride..

Again, this is a decision the LLVM community needs to make based on its own cost/benefit analysis. We often don't start new projects with mature code -- in fact, doing that is an unusual case.

> 
> All the positive interactions and feedback shouldn't be exclusive
> with
> SE being a project. Tying them together doesn't seem like the right
> approach.
> 
> It seems like "I want it MY way or I won't play"...

Perhaps a fundamental difference in our opinions here comes from the fact that I've not read the interaction in this way.

Thanks again,
Hal

> How about we revisit this in a month? If in 1 month other codes are
> using SE we can discuss the benefits...
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory