[llvm-dev] [RFC] GitHub Survey - Please review

[Chris Bieneman via llvm-dev]

> On Aug 19, 2016, at 11:04 AM, Renato Golin <renato.golin at linaro.org> wrote:
> 
> Hi Chris,
> 
> Bear in mind that the more questions we have, the harder it will be to
> interpret the results. If we have 20+ questions, it'll be impossible
> to understand anything.
> 
> Also, the multiple choice questions are meant as a guide to understand
> "how many" people fall into one or another category, while the free
> text ones are meant to complement and give technical reasons for their
> answers.
> 
> So, we should focus our multiple choice questions on divisive topics
> and let everything else to the free text-fields.
> 
> 
> On 19 August 2016 at 18:06, Chris Bieneman <beanz at apple.com> wrote:
>> (1) Which project(s) do you contribute to?
>> (2) Which project(s) do you regularly use?
> 
> I've added these two as one. I know they're slightly different, but so
> will be the the answer to the first question, which will work to
> disambiguate this one.
> 
> 
>> (3) How often do you bisect LLVM with one or more subproject?
> 
> I understand that this is a contentious issue around the git move, but
> we should focus on the bigger picture, which is day to day usage as
> well as infrastructure.

I disagree. I think this is a really important thing to understand how people interact with the SCM system. I think one part of the benefit of doing a survey is that we can gather user data about how common specific workflows are.

> 
> 
>> (4) Do you use any of the llvm.org projects without LLVM or with out-of-sync
>> LLVM (i.e. trunk libunwind with an old LLVM)?
> 
> This looks very specific to me, I'm trying to avoid side questions
> here and let people write up on the free text areas what their usage
> is.

Fair enough. I do think that relying on the write-up fields is potentially difficult. If everyone who contributed to LLVM projects in the last year responds to the survey you’re talking about somewhere on the order of 500 responses. More if non-contributors respond (and I hope they do). Weeding through lots of write-in data could be come a monumental task. I think it is better to have more yes/no and multiple choice questions to help collate the data. Admittedly wording them neutrally will be a challenge, but I think it is something we should do.

> 
> 
>> (5) In which ways do you get LLVM sources from LLVM.org?
>>   (a) SVN
>>   (b) llvm.org Git mirrors
>>   (c) Git-SVN
>>   (d) GitHub Git mirrors
>>   (e) Other
> 
> I don't think that previous usage is relevant. It may be relevant to
> the people doing it and to their responses on how hard it will be, but
> this should be encoded in the other questions. Some of that already
> is.

I also disagree on this. I think it is useful to know which workflows are common in the community. It will help make informed decisions about the direction of the infrastructure.

> 
> 
>> (6) Do you, or an organization you are affiliated with, maintain tooling or
>> infrastructure that interacts with llvm.org and is not public?
> 
> This is a topic for the free-text fields.

I feel less invested in this question than in some of the others, because I think we know that this is a fairly common situation (especially with the large corporate contributors), but I still think relying on free-text fields for questions that can be yes/no or multiple choice will degrade our ability to process the information.

In general I think there is a sweet-spot on the length of the survey. We don’t want to go overboard with too many questions, but I think that relying on text fields could turn out to make the data very difficult to process if we get a lot of respondents.

-Chris

> 
> cheers,
> --renato