[llvm-dev] RFC Process

[Renato Golin via llvm-dev]

On Tue, 30 Oct 2018 at 20:13, Kristina Brooks via llvm-dev
<llvm-dev at lists.llvm.org> wrote:
> Just wanted to voice my opinion on the matter, I think mailing lists
> can be fine for RFCs but very often they get ignored or simply get one
> or two replies disagreeing with the idea, which basically makes for the
> whole consensus.

Hi Kristina,

I appreciate this is what it looks like (and I've been foul of
assuming agreement was consensus quite a few times before!), but
ultimately, we're down to either noise or bureaucracy.

> Currently it feels like the only viable way of actually proposing something
> is having a diff up on Phabricator for review, with an already finished
> implementation, which is entirely not the point of RFCs. I'm not sure I have
> a better suggestion but possibly a section on Phabricator for RFC reviews
> similar to pre-commit reviews would be a nicer alternative.

I agree Phab is not the right place for RFCs per se, but they're quite
helpful to RFCs with examples of implementation, exploring
alternatives, etc.

> At least that has generally been my experience, with using the mailing lists
> for this and a rather frustrating one since drafting a proposal takes time
> and it's frustrating when it just gets almost no attention.

I think we all share the same pain, honestly.

> I think Phab
> as an alternative would be an excellent idea if it's possible to make it
> separate from the pre-commit review section but in the same format, and allow
> for more discussion to take place.

We have been unofficially using docs/Proposals for more elaborate
proposals with some success. Both the GitHub and the VPlan proposals
got enough traction in the end, but they're also massive amounts of
work and asking every little RFC to do the same would dilute the
benefits, I think.

Back to my first point, I have proposed a few times that we use a
system (I think last time it was Bugzilla a long time ago) to do RFCs
in an organised way, copying the right people, etc. The problem with
that is two fold: first, it segregates to the people you know who to
copy, and may miss entirely people that would have a lot more to say,
and second, it creates the duty to review something you may not have
expertise at all, or has moved on and can't comment, or didn't receive
the email in the first place.

IMHO, trying to fix poor reviewed RFCs with a system would not lead to
*more* reviews. It wouldn't fix the limbo problem.

Now, consider the other side of the coin. When I review RFCs that I
have an opinion, I go as far as I can on my knowledge and experience
to give as broad review as possible. But sometimes that's not enough
because I just don't know enough about that one RFC to be useful. I
may very well not like the idea, from experience. But it could be just
because I'm biased or ignorant, and I'd be more than happy for other
more experienced people to disagree with me and actually review the
RFC.

But there's this assumption that, if someone is looking at an RFC, I
don't need to look at that, too. We all have busy schedules, so we
tend to pick our battles. But then, you're stuck with my horrible
review and give up. This is not good.

The solution to both problems, I think, is the same: noise.

If you don't agree with the reviewers on their negative comments, copy
more people into the thread, ask on IRC who's the actual expert,
include them in, make noise, be heard. Also, get opinions from a
diverse audience, from back-ends to front-ends, from companies to
academia, etc. Bug people on LLVM meetings, CGO, workshops. If you can
get such a broad range of views and none of them agree with you, them
quite likely your RFC needs some refactoring...

The same logic is true on the other side. If a reviewer doesn't have
the expertise and feels out of depth, be candid about your
shortcomings. Check the list history for people on that area, check
code owners, bugzilla entries, buildbot owners. We don't have special
status for code owners exactly because it scales better, and if we all
make enough signal when it's important, then we might just get over
the noise and get a meaningful review.

The way we stand out of the crowd is by prefixing emails/reviews with
RFC, which lets people setup filters, but copying the right people,
even if after you send it, is the sure way of getting attention.

But, I stand by my comments on bias and ignorance. I'm happy to be
proven wrong and have an RFC system that works. Feel free to copy me
on that RFC. :)

cheers,
--renato

PS: Hal's test-suite RFC is a long term one and that's why it's in
Proposals. It's a request to build a list, so that people can start
from that list. Other RFCs on actual code (including memory models,
scalable vectors, parallel IR) usually gather much more attention,
even though they're still very high level.