[LLVMdev] Atomic Operation and Synchronization Proposal v2

[Chandler Carruth]

On 7/12/07, Dan Gohman <djg at cray.com> wrote:
> On Thu, Jul 12, 2007 at 10:06:04AM -0500, David Greene wrote:
> > On Thursday 12 July 2007 07:23, Torvald Riegel wrote:
> >
> > > > The single instruction constraints can, at their most flexible, constrain
> > > > any set of possible pairings of loads from memory and stores to memory
> > >
> > > I'm not sure about this, but can we get issues due to "special" kinds of
> > > data transfers (such as vector stuff, DMA, ...?). Memcpy implementations
> > > could be a one thing to look at.
> > > This kind of breaks down to how universal you want the memory model to be.
> >
> > Right.  For example, the Cray X1 has a much richer set of memory ordering
> > instructions than anything on the commodity micros:
> >
> > http://tinyurl.com/3agjjn
> >
> > The memory ordering intrinsics in the current llvm proposal can't take
> > advantage of them because they are too coarse-grained.
>
> I guess the descriptions on that page are, heh, a little terse ;-).

A bit. ;] I was glad to see your clarification.

> The
> Cray X1 has a dimension of synchronization that isn't covered in this
> proposal, and that's the set of observers need to observe the ordering.
> For example you can synchronize a team of streams in a multi-streaming
> processor without requiring that the ordering of memory operations be
> observed by the entire system. That's what motivates most of the variety
> in that list.

This is fascinating to me, personally. I don't know how reasonable it
is to implement directly in LLVM, however, could a codegen for the X1
in theory establish if the "shared memory" was part of a stream in a
multi-streaming processor, and use those local synchronization
routines? I'm not sure how reasonable this is. Alternatively, to
target this specific of an architecture, perhaps the LLVM code could
be annotated to show where it is operating on streams, versus across
processors, and allow that to guide the codegen decision as to which
type of synchronization to utilize. As LLVM doesn't really understand
the parallel implementation the code is running on, it seems like it
might be impossible to build this into LLVM without it being
X1-type-system specific... but perhaps you have better ideas how to do
such things from working on it for some time?

>
> There's one other specific aspect I'd like to point out here. There's an
> "acquire" which orders prior *scalar* loads with *all* subsequent memory
> accesses, and a "release" which orders *all* prior accesses with subsequent
> *scalar* stores. The Cray X1's interest in distinguishing scalar accesses
> from vector accesses is specific to its architecture, but in general, it
> is another case that motivates having more granularity than just "all
> loads" and "all stores".

This clarifies some of those instructions. Here is my thought on how
to fit this behavior in with the current proposal:

You're still ordering load-store pairings, there is juts the added
dimensionality of types. This seems like an easy extension to the
existing proposal to combine the load and store pairings with a type
dimension to achieve finer-grained control. Does this make sense as an
incremental step from your end with much more experience comparing
your hardware to LLVM's IR?

>
> Overall though, I'm quite happy to see that the newest revision of the
> proposal has switched from LLVM instructions to LLVM intrinsics. That
> will make it easier to experiment with extensions in the future. And
> having the string "atomic" right there in the names of each operation
> is very much appreciated :-).

The atomic in the name is nice. It does make the syntax a bit less
elegant, but it'll get the ball rolling faster, and thats far more
important! Thanks for the input, and I really love the X1 example for
a radically different memory model from the architectures LLVM is
currently targeting.

-Chandler Carruth


>
> Dan
>
> --
> Dan Gohman, Cray Inc.
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>