[llvm-dev] [PITCH] Improvements to LLVM Decision Making

Wed Jan 15 02:04:15 PST 2020

Hi Chris,

I experience a fair share of problems when it comes to code reviews and
change requests so I appreciate any improvement we can make here.

On 01/14, Chris Lattner via llvm-dev wrote:
> Numerous people have been bringing up challenges with consensus driven
> decision making in the LLVM community.  After considering this and
> seeing the frustrations it is causing many people, I think we should
> make a formal process change to help improve decision making going
> forward.

Improving is always a good idea.

> Here is the outline of the draft proposal
> <https://gist.github.com/lattner/e3679998a7609c99b1243f09d30f0132>.
> Caveats: this is itself just a draft proposal to inspire discussion
> not a mandate - I welcome feedback, thoughts and concerns, ideas of
> how to make this better, or better ideas/approaches overall on how we
> can level up the community.
> 
> Please let me know what you think, thanks!

First thought: This should not be a gist on github. Maybe it should be
on phabricator or at least in an email so it is easier to respond to it.

I'll try to inline the gist below so I can respond to the points.

---

> "It isn't clear how to propose some changes in the first place, and it
> is often unclear who the decision makers are."

I feel that patches and RFCs are well established *and documented* [1,2]
ways to propose changes. In addition, the *-dev lists, IRC, etc. do
provide ways to clear uncertainties. Adding more documentation on this
can obviously be helpful. Also, we are already improving the existing
documentation [0].

That said, we need to differentiate in more detail what the problems
actually are. Do people not find the documentation? Is the documentation
unclear? Are people waiting for "a decision maker" to step in or afraid
to ask questions? Are questions not answered? ...

> "It isn't clear what mailing lists to send such proposals to: llvm-dev
> gets a lot of traffic and many people don’t read it."

I argue that people that do participate in the decision making (wording
took from the previous point) already do, or at least should, follow
llvm-dev.
I am not quite sure why it is unclear what mailing list is the right one
given that we have one *-dev mailing list per subproject and on llvm-dev,
often the first points-of-contact, people are referred to the right one.

> "It is hard to know whether something is a serious proposal that you
> must take seriously, and so it is easy to miss important changes
> because you don't have time to review everything. Even though you
> technically had a chance to participate, you can end up surprised when
> some change goes into effect."

I'm unsure what kinds of proposal are supposed to be "not serious" and
who is supposed to be making that decision.

> "Sometimes people chime in late with dissent after a decision has been
> apparently made: this can be frustrating to people who need a decision
> made, because they aren't sure how to proceed."

It is unclear to me how the proposal helps on this front. Could you
elaborate?

> "Sometimes people express a loud voice on discussion threads even if
> they aren't active contributors, and they can derail discussions.
> There is no "moderator" for these discussions."

With the caveat of finding the moderator (as mentioned below), this
makes sense. We probably/hopefully don't need a moderator (for this
reason) on many discussions but it might certainly help if people step
up if a discussion is derailed (for any reason and by anyone).

> "The initial discussion phase of a proposal can have lots of back and
> forth discussions to shape a idea, and the eventual proposal can have
> significant changes from that initial review. It can be difficult to
> discern what feedback from the initial review is addressed or ignored
> in the final rounds of discussions."

Yes. I am not sure how the proposed solution remedies the problem
though. Could you elaborate?

> "Complex changes (e.g. the relicensing project) sometimes take many
> rounds of iteration, and it can be easy to lose track of where the
> proposal is and what the history was."

This is certainly true. The proposed solution (with moderators and
rounds) is probably implementable and reasonable for "complex changes".

> "Occasionally, Code Owners may be faced with a technical decision and
> not be sure what to do, particularly for highly impactful design
> decisions - e.g. for core LLVM IR changes. It could be useful to have
> a formal escalation process for these decisions."

TBH, I always thought "code owner" is not a "decision making" but rather
an "administrative" title. The person can be part of the decision
making, but the role of "code owner" does not imply special privileges,
only tasks, e.g., making sure reviews are done by the right people.

> I recommend that we add a process similar to (but adapted and changed
> where it makes sense) the Swift Evolution process. This process was
> designed to help guide decision making for high impact language and
> standard library changes to the Swift language. It was inspired by
> similar processes in other language communities (e.g. the Rust
> Language RFC process, the Python PEP process, etc) and has worked well
> in that context - it stands to reason that a variant should work well
> for LLVM as well.

I tried to determine how Rust RFCs (and Python PEPs) influenced the
proposed Swift Evolution process but from the history of the linked
pages (in the gist) that was not clear. Could you elaborate on that?

> The solution entails several parts. First, the process should be
> written down!

Agreed!

> This written process should include things like:
> 
>     An informal "pitch" phase to help collect requirements and shape
>     the ultimate form of an idea, but which can be ignored by people
>     who aren't interested in following all of the details of a
>     discussion.

How is this different form the discussion that happens after an initial
RFC is send?

People already ignore it if they are not interested in the details.  If
people chime in late, as mentioned in the problems above, this will not
help, right? I mean, if the pitch phase is done and then people start to
chime it starts again.  This can have any reason, they are late, they
want to see if it is really "a serious proposal", or they just want to
wait until the first round of discussion changed the proposal to start
the second round.

>     A new mailing list (or Discourse channel) dedicated to formal
>     proposal review. This would be relatively low volume, allowing
>     effectively everyone to follow it. This could be named something
>     like "llvm-proposals".

1) We have already 30+* Discourse channels. Having so many, and one
   more, makes it harder to actually monitor them. Additionally, people
   that do not have Discourse are already excluded from the discussion.
   (I feel this had/has the opposite effect it was supposed to have.)
2) Arguably you could filter *-dev lists for the tag RFC instead of
   having a new mailing list^. People sending RFCs send without the tag
   can be asked to send it again with the tag. 
3) This would only be low-volume if you do not count the
   responses/discussion.

* I haven't counted them but I am probably close with my estimate.
^ We have a lot already which can be, as implicit mentioned above,
  confusing.

>     A written proposal template, which provides guidelines for how to
>     write a proposal. This proposal is written in an example template
>     we can use, and the template can evolve over time.

I'm in favor.

>     A new directory on Github that serves as the archive for official proposals.

I don't see how that helps so I'd like to ask you to elaborate why this
is not yet another place one has to look for information.

>     A review manager who helps shepherd the discussion in official
>     rounds of review (which are time bound, e.g. to a week or two).

Could you elaborate on how these review managers are determined?

>     A decision making body that evaluates the results, asking for
>     additional discussions (with advice for how to change the proposal
>     in the next round), and that ultimately approves or denies a
>     proposal. This body should aim to get the community to consensus
>     whenever possible, but can help split decisions in the case of
>     extreme ambiguity when overwise all hope is lost. Denied proposals
>     can of course be re-run if a pertinent situation changes or when
>     revised

Could you elaborate on how these "decision making bodies" are
determined?

Thanks for initiating this,
  Johannes

[0] https://reviews.llvm.org/D71916
[1] http://llvm.org/docs/DeveloperPolicy.html#code-reviews
[2] https://www.llvm.org/docs/Contributing.html
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 228 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20200115/e441257b/attachment.sig>