[llvm-dev] GitHub Survey?

[Chris Bieneman via llvm-dev]

> On Sep 9, 2016, at 12:03 PM, Renato Golin via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> On 9 September 2016 at 19:05, Chris Bieneman <cbieneman at apple.com> wrote:
>> I think having the survey contain a question on which solution the respondent prefers is good, but I feel it is very limited.
> 
> Well, the current survey is more than just one question…

Sorry, I didn’t mean to imply that was the only question.

> 
> 
>> In general I believe there are three possible decisions that could come from this. Either we go with one of the two proposals, or we change nothing. Knowing how many users are already using Git speaks to the impact of a transition from SVN to Git regardless of which Git proposal, and is a useful bit of data to collect.
> 
> We have three specific questions as to what the impact of the
> transition (one for each option short term plus one for long term).
> Why doesn't that cover your proposal?

There is a difference between asking “How much does this impact you?” verses getting data on what about it causes impact. For example someone who thinks a transition will greatly impact them but also is already using git may have specific interesting concerns.

> 
> 
>> Knowing this allows the decision maker(s) to weight results based on the opinions and impacts on contributors differently from the opinions and impacts on down-stream users. While I certainly don't think we should disregard downstream users, this decision will disproportionately impact contributors, so we need to take that into account.
> 
> We already have that *exact* question.
> 
> It's a multiple choice questions where you check all projects that you
> work on. The pie chart means nothing to the GitHub survey per se, but
> the association of that information with the repo choice and the
> impact will be very valuable.
> 
> For example, if most libc++ developers prefer option X while core LLVM
> developers prefer Y.
> 
> What would you suggest in addition to what's already there?

Where I think the current survey is lacking is the ability to differentiate uses from contributions. Probably the only thing we need to add to the survey to cover this is either a clear statement that the email provided should be the email tied to their SVN account, or a request for SVN username if the person has commit access.

> 
> 
> 
>> In addition to that I believe we should actually provide a section of the survey specifically for questions that can inform the specific proposals, and improve them.
> 
> Right. I steered away from that on purpose, but I guess we can add one
> free text field for each impact question, and mention that people
> should laid out why this or that option is not good, and how to make
> them good, in the case where it's chosen.
> 
> 
>> For example, the mono-repo proposal currently lacks firm details on a few things, which might benefit from survey answers. For example, which projects should be included in the mono-repo, or are per-project git mirrors important.
> 
> Indeed, this is currently missing. Would that be enough to cover this
> in the free-text I mention above?

As I’ve said in the past analyzing data from free text fields will be unwieldy. By my count, we had 473 contributors across clang, clang-tools-extra, compiler-rt, libcxx, libcxxabi, libunwind, lld, lldb, and llvm in the last year*. In an ideal world we’d get 100% response to the survey plus additional responses from downstream users who aren’t contributors. Analyzing free-text fields for hundreds of respondents to get data that could come from very simple questions seems less than ideal.

All that aside, none of this may matter. It largely depends on who is making the decision and how they are making the decision. It might be nice to get some consensus around that as a starting point.

-Chris

*Data gathered by running:
for repo in $(find . -name .git); do git --git-dir=$repo log --format=%an --since='1 year ago'; done | sort | uniq -c | wc -l

> 
> cheers,
> --renato
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev