[llvm-dev] PSA: debuginfo-tests workflow changing slightly

Tue Nov 21 21:29:10 PST 2017

Definitely welcome to look into it, but unless you have physical access to
one of these buildbots, I don't think we should commit anything else
speculatively without having someone who does have physical access first
confirm it.

On Tue, Nov 21, 2017 at 7:25 PM Don Hinton <hintonda at gmail.com> wrote:

> I sorta enjoy debugging stuff like this, so if you don't mind, I'll dig
> into it once I get a chance -- traveling so, my access is a bit sketchy
> right now.
>
> I'll see if I can grab the logs and let you know if I find anything
> interesting.
>
> On Tue, Nov 21, 2017 at 7:04 PM, Zachary Turner <zturner at google.com>
> wrote:
>
>> That change was added specifically to workaround a failure in a previous
>> version of the commit.  I think as chandler says it's pretty much
>> impossible for me to make any progress unless someone whose build bot is
>> failing actually sits down, applies the patch, makes it work, and then
>> sends me back the result.
>>
>> On Tue, Nov 21, 2017 at 7:03 PM Don Hinton <hintonda at gmail.com> wrote:
>>
>>> Hi Zackary,
>>>
>>> Did you see my followup to you cfe-commits rollback:
>>>
>>>
>>> http://lists.llvm.org/pipermail/cfe-commits/Week-of-Mon-20171120/210212.html
>>>
>>> I think you can just remove the change to cfe/trunk/test/CMakeLists.txt
>>> and it should work just fine.
>>>
>>> hth...
>>> don
>>>
>>> On Tue, Nov 21, 2017 at 12:00 PM Zachary Turner via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>> Just an update,
>>>>
>>>> At this point I've submitted and reverted this probably 6+ times, and I
>>>> don't have time to be blocked by this anymore.
>>>>
>>>> I'm looking into alternatives for now, including ways of creating a new
>>>> top-level git repo such as https://git.llvm.org/git/codeview-tests.git/
>>>> .
>>>>
>>>> I wish there were another way, but at this point I've been working on
>>>> appeasing this for close to 2 months and I need some control over the bits
>>>> that determine whether I'm able to make forward progress.
>>>>
>>>> If and when someone on the Apple side has some time to make this work
>>>> with their build infrastructure, I will be happy to revisit.
>>>>
>>>> On Mon, Nov 13, 2017 at 4:44 PM Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Great!  It's close to the end of the day, so I'll submit tomorrow to
>>>>> make sure everything has a chance to go fully green again to ensure I get
>>>>> failure emails if it breaks.
>>>>>
>>>>> On Mon, Nov 13, 2017 at 4:43 PM Adrian Prantl <aprantl at apple.com>
>>>>> wrote:
>>>>>
>>>>>> I can confirm that that fixes the issue!
>>>>>>
>>>>>> — adrian
>>>>>>
>>>>>>
>>>>>> On Nov 13, 2017, at 4:38 PM, Zachary Turner <zturner at google.com>
>>>>>> wrote:
>>>>>>
>>>>>> Yea I also just found it.  Try adding this code in the bottom of
>>>>>> debuginfo-tests/lit.cfg.py
>>>>>>
>>>>>> lit.util.usePlatformSdkOnDarwin(config, lit_config)
>>>>>>
>>>>>>
>>>>>> On Mon, Nov 13, 2017 at 4:38 PM Adrian Prantl <aprantl at apple.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Ha! Found it. *Somebody* is setting an SDKROOT variable in the
>>>>>>> environment. Can you find the code that would do this?
>>>>>>>
>>>>>>> — adrian
>>>>>>>
>>>>>>> > On Nov 13, 2017, at 4:30 PM, Adrian Prantl via llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>> >
>>>>>>> > Yes I can reproduce this locally. It looks like we are not passing
>>>>>>> an -isysroot (pointing to the SDK) to clang but it isn’t clear what lit
>>>>>>> magic would expand this.
>>>>>>> >
>>>>>>> > -- adrian
>>>>>>> >
>>>>>>> >> On Nov 13, 2017, at 3:30 PM, Zachary Turner <zturner at google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Yea I'm preparing a revert right now.  Does it happen for you
>>>>>>> when you run debuginfo-tests locally?
>>>>>>> >>
>>>>>>> >> On Mon, Nov 13, 2017 at 3:28 PM Adrian Prantl <aprantl at apple.com>
>>>>>>> wrote:
>>>>>>> >> Since this is causing all of our internal CI to back up, could
>>>>>>> you please revert your two changes, so we can make sure that they were
>>>>>>> actually responsible, and can work on a fix for this? Let me know how we
>>>>>>> can help investigate this.
>>>>>>> >>
>>>>>>> >> -- adrian
>>>>>>> >>
>>>>>>> >>
>>>>>>> >>> On Nov 13, 2017, at 3:25 PM, Adrian Prantl via llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>> >>>
>>>>>>> >>> The first build where a test fails with similar symptoms has
>>>>>>> just one blamelist entry:
>>>>>>> >>>
>>>>>>> >>>
>>>>>>> http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40391/
>>>>>>> >>>
>>>>>>> >>>     • Update test_debuginfo.pl script to point to new tree
>>>>>>> location. (detail/ViewSVN)
>>>>>>> >>> by zturner
>>>>>>> >>> -- adrian
>>>>>>> >>>
>>>>>>> >>>> On Nov 13, 2017, at 3:21 PM, Zachary Turner <zturner at google.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> On the other hand this file hasn't changed recently, but I have
>>>>>>> no way to test this as it uses the LLDB code path, which only runs on OSX.
>>>>>>> >>>>
>>>>>>> >>>> On Mon, Nov 13, 2017 at 3:19 PM Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>> I might be missing something, but this doesn't look like me?
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40478/consoleFull#-42777206a1ca8a51-895e-46c6-af87-ce24fa4cd561
>>>>>>> >>>>
>>>>>>> >>>> PASS: debuginfo-tests :: dbg-arg.c (34886 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: ctor.cpp (34887 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: ctor.cpp (34888 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: aggregate-indirect-arg.cpp (34889 of
>>>>>>> 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: aggregate-indirect-arg.cpp (34890 of
>>>>>>> 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: dbg-arg.c (34891 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: asan.c (34892 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: asan.c (34893 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: asan-blocks.c (34894 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: asan-blocks.c (34895 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: nested-struct.cpp (34896 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: nested-struct.cpp (34897 of 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: forward-declare-class.cpp (34898 of
>>>>>>> 40729)
>>>>>>> >>>> PASS: debuginfo-tests :: forward-declare-class.cpp (34899 of
>>>>>>> 40729)
>>>>>>> >>>> FAIL: debuginfo-tests :: foreach.m (34900 of 40729)
>>>>>>> >>>>
>>>>>>> >>>> ******************** TEST 'debuginfo-tests :: foreach.m' FAILED
>>>>>>> ********************
>>>>>>> >>>>
>>>>>>> >>>> Script:
>>>>>>> >>>> --
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang
>>>>>>> --target=x86_64-apple-darwin15.6.0 -O0 -g
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m
>>>>>>> -c -o
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang
>>>>>>> --target=x86_64-apple-darwin15.6.0
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o
>>>>>>> -o
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.out
>>>>>>> -framework Foundation
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/
>>>>>>> >>>> test_debuginfo.pl
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.out
>>>>>>> >>>> --
>>>>>>> >>>> Exit Code: 1
>>>>>>> >>>>
>>>>>>> >>>> Command Output (stdout):
>>>>>>> >>>> --
>>>>>>> >>>> Debugger output was:
>>>>>>> >>>> imported lldb from:
>>>>>>> "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/Python"
>>>>>>> >>>> error: foreach.m.tmp.out debug map object file
>>>>>>> '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o'
>>>>>>> has changed (actual time is 0x5a0a2528, debug map time is 0x5a0a2526) since
>>>>>>> this executable was linked, file will be ignored
>>>>>>> >>>> >  break 25
>>>>>>> >>>> SBBreakpoint: id = 1, file = '', line = 25, exact_match = 0,
>>>>>>> locations = 0
>>>>>>> >>>> >  r
>>>>>>> >>>> success
>>>>>>> >>>> >  po thing
>>>>>>> >>>>  = <could not resolve type>
>>>>>>> >>>> > quit
>>>>>>> >>>>
>>>>>>> >>>> --
>>>>>>> >>>> Command Output (stderr):
>>>>>>> >>>> --
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/tests/foreach.m:11:11:
>>>>>>> error: expected string not found in input
>>>>>>> >>>>
>>>>>>> >>>> // CHECK: aaa
>>>>>>> >>>>           ^
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.gdb.output:1:1:
>>>>>>> note: scanning from here
>>>>>>> >>>> imported lldb from:
>>>>>>> "/Applications/Xcode.app/Contents/SharedFrameworks/LLDB.framework/Versions/A/Resources/Python"
>>>>>>> >>>> ^
>>>>>>> >>>>
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.gdb.output:2:211:
>>>>>>> note: possible intended match here
>>>>>>> >>>> error: foreach.m.tmp.out debug map object file
>>>>>>> '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/tools/clang/test/debuginfo-tests/Output/foreach.m.tmp.o'
>>>>>>> has changed (actual time is 0x5a0a2528, debug map time is 0x5a0a2526) since
>>>>>>> this executable was linked, file will be ignored
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>> On Mon, Nov 13, 2017 at 3:17 PM Adrian Prantl <
>>>>>>> aprantl at apple.com> wrote:
>>>>>>> >>>> It looks like the bots are still red?
>>>>>>> >>>>
>>>>>>> >>>> — Adrian
>>>>>>> >>>>
>>>>>>> >>>>
>>>>>>> >>>>> On Nov 10, 2017, at 3:14 PM, Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> Wasn't quite fixed, but it got a lot further this time.  This
>>>>>>> time there was still an issue in the test_debuginfo.pl script
>>>>>>> regarding a hardcoded path to the llgdb.py script.  I think I never
>>>>>>> encountered this locally because this codepath only happens on Darwin, and
>>>>>>> I was testing on Linux.
>>>>>>> >>>>>
>>>>>>> >>>>> (As an aside, ugh... Perl...)
>>>>>>> >>>>>
>>>>>>> >>>>> Regardless, this should be fixed in r317949, and hopefully
>>>>>>> that's the last of the issues.  I have to run for a couple of hours, but I
>>>>>>> can check on this again in a bit.  But I strongly suspect it will be fixed
>>>>>>> now.
>>>>>>> >>>>>
>>>>>>> >>>>> On Fri, Nov 10, 2017 at 2:51 PM Adrian Prantl <
>>>>>>> aprantl at apple.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>>> On Nov 10, 2017, at 2:50 PM, Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> I checked in a fix for that already, sorry for the trouble.
>>>>>>> I’m waiting for it to cycle
>>>>>>> >>>>>
>>>>>>> >>>>> awesome. Thanks!
>>>>>>> >>>>>
>>>>>>> >>>>> -- adrian
>>>>>>> >>>>>
>>>>>>> >>>>>> On Fri, Nov 10, 2017 at 2:49 PM Adrian Prantl <
>>>>>>> aprantl at apple.com> wrote:
>>>>>>> >>>>>> It looks like this broke green dragon:
>>>>>>> >>>>>>
>>>>>>> >>>>>>
>>>>>>> http://green.lab.llvm.org/green/job/clang-stage1-configure-RA/40383/console
>>>>>>> >>>>>>
>>>>>>> >>>>>> llvm-lit:
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/projects/libcxx/utils/libcxx/test/config.py:173:
>>>>>>> note: Adding environment variables: {'DYLD_LIBRARY_PATH':
>>>>>>> '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/./lib',
>>>>>>> 'LIBCXX_FILESYSTEM_DYNAMIC_TEST_ROOT':
>>>>>>> '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/projects/libcxx/test/filesystem/Output/dynamic_env'}
>>>>>>> >>>>>>
>>>>>>> >>>>>> llvm-lit:
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/llvm/config.py:332:
>>>>>>> note: using clang:
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/clang-build/bin/clang
>>>>>>> >>>>>> llvm-lit:
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/util.py:379:
>>>>>>> note: using SDKROOT:
>>>>>>> '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk'
>>>>>>> >>>>>>
>>>>>>> >>>>>> llvm-lit:
>>>>>>> /Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/TestingConfig.py:101:
>>>>>>> fatal: unable to parse config file
>>>>>>> '/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/
>>>>>>> lit.cfg.py', traceback: Traceback (most recent call last):
>>>>>>> >>>>>>   File
>>>>>>> "/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/utils/lit/lit/TestingConfig.py",
>>>>>>> line 88, in load_from_path
>>>>>>> >>>>>>     exec(compile(data, path, 'exec'), cfg_globals, None)
>>>>>>> >>>>>>   File
>>>>>>> "/Users/buildslave/jenkins/workspace/clang-stage1-configure-RA/llvm/tools/clang/test/debuginfo-tests/
>>>>>>> lit.cfg.py", line 36, in <module>
>>>>>>> >>>>>>     config.test_source_root =
>>>>>>> os.path.join(config.debuginfo_tests_src_root, 'tests')
>>>>>>> >>>>>> AttributeError: TestingConfig instance has no attribute
>>>>>>> 'debuginfo_tests_src_root'
>>>>>>> >>>>>>
>>>>>>> >>>>>> FAILED: CMakeFiles/check-all
>>>>>>> >>>>>>
>>>>>>> >>>>>> -- adrian
>>>>>>> >>>>>>
>>>>>>> >>>>>> > On Nov 10, 2017, at 1:00 PM, Zachary Turner via llvm-dev <
>>>>>>> llvm-dev at lists.llvm.org> wrote:
>>>>>>> >>>>>> >
>>>>>>> >>>>>> > This is in as of r317925.  I'm keeping an eye out for
>>>>>>> failure notifications.  I may or may not need help diagnosing if something
>>>>>>> does go wrong (although I'm keeping my fingers crossed)
>>>>>>> >>>>>> >
>>>>>>> >>>>>> > On Thu, Nov 9, 2017 at 4:05 PM Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>> > Since it's towards the end of the day already, I'll put
>>>>>>> this in tomorrow morning around 9 or 10, to make sure I'm around to fix
>>>>>>> anything that arises (or revert).
>>>>>>> >>>>>> >
>>>>>>> >>>>>> >
>>>>>>> >>>>>> >
>>>>>>> >>>>>> > On Thu, Nov 9, 2017 at 2:53 PM Mike Edwards <
>>>>>>> medwards at apple.com> wrote:
>>>>>>> >>>>>> > Hi Zach,
>>>>>>> >>>>>> > Thanks for doing this extra work to make this lower impact
>>>>>>> for the rest of us.  Let’s give it a try and see what happens.
>>>>>>> >>>>>> >
>>>>>>> >>>>>> > -Mike
>>>>>>> >>>>>> >
>>>>>>> >>>>>> >
>>>>>>> >>>>>> >
>>>>>>> >>>>>> >> On Nov 9, 2017, at 13:37, Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >> Hi all, I think I've addressed all the concerns here, and
>>>>>>> I believe there should be no immediate impact to the current workflow.
>>>>>>> with that said, I plan to commit this either later today or early tomorrow
>>>>>>> if there are no other concerns.
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >> On Tue, Nov 7, 2017 at 12:19 PM Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>> >> I tested this out, and AFAICT nothing will change.  It
>>>>>>> will continue to just work if you have it checked out under clang/tests.
>>>>>>> It's a bit hard to construct this configuration locally since it requires
>>>>>>> moving some files around, and applying half of a CL here and half of a CL
>>>>>>> there.  But, AFAICT it works.
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >> I'm happy to send you some patches if you want to try them
>>>>>>> locally and confirm.
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >> I'd like to print out a CMake warning if it detects the
>>>>>>> tree under clang/test and just mention that the workflow is deprecated.
>>>>>>> Any objections?
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >> On Mon, Nov 6, 2017 at 1:49 PM Mike Edwards <
>>>>>>> medwards at apple.com> wrote:
>>>>>>> >>>>>> >> Thank you Zach.
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >>> On Nov 6, 2017, at 13:37, Zachary Turner <
>>>>>>> zturner at google.com> wrote:
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>> I’m going to spend a little time seeing if i can make the
>>>>>>> change invisible to the bots so they will continue to work as they do
>>>>>>> today.  Will report back after I’ve explored that a bit
>>>>>>> >>>>>> >>> On Mon, Nov 6, 2017 at 1:35 PM Mike Edwards <
>>>>>>> medwards at apple.com> wrote:
>>>>>>> >>>>>> >>>> I'm honestly not opposed to this idea.  It just seems a
>>>>>>> shame to do this for purely logistical reasons if most people agree that
>>>>>>> the "right" place for debuginfo-tests is outside of the clang tree.
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>> I totally understand what you are saying here and will
>>>>>>> just add that sometimes being part of a larger community means being
>>>>>>> willing to do things, sometimes, not exactly the “right” way, due to
>>>>>>> logistical reasons.  I am not opposed to what you would like to do, I’m
>>>>>>> just furrowing my brow at the timeframe in which to do it.
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>>>
>>>>>>> >>>>>> >>>> That said, I'd still like to hear from ChrisM and MikeE
>>>>>>> about why it will take so long, because on the surface it seems like a
>>>>>>> low-impact move.
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>> Past experience has taught me, anything I think is going
>>>>>>> to be simple and quick to fix, rarely ever turns out that way.  While there
>>>>>>> will be a significant amount of work to change the way our bots work here
>>>>>>> at Apple, the work is not impossible to accomplish.  Given the choice, I
>>>>>>> would of course prefer an approach such as Paulr has suggested.  The
>>>>>>> ability to run things in parallel for a time provides for a much lower
>>>>>>> impact change on the entire community.  I think this approach may also give
>>>>>>> us some time to decide where the debuginfo-test should fit in the new
>>>>>>> mono-repo.  It would be a bummer to do the work necessary to make this
>>>>>>> change, only to discover we would have to do it differently in the not too
>>>>>>> distant future to accommodate the new mono-repo.
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>>  Zach, I do not want to be a blocker here.  I just want
>>>>>>> to make sure we have explored all of the options to make sure we are not
>>>>>>> missing a lower impact approach.  I also want to make sure we are not doing
>>>>>>> something that could wait until we migrate to the mono-repo next year.
>>>>>>> >>>>>> >>>
>>>>>>> >>>>>> >>> Thanks,
>>>>>>> >>>>>> >>> Mike
>>>>>>> >>>>>> >>
>>>>>>> >>>>>> >
>>>>>>> >>>>>> > _______________________________________________
>>>>>>> >>>>>> > LLVM Developers mailing list
>>>>>>> >>>>>> > llvm-dev at lists.llvm.org
>>>>>>> >>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>> >>>>>>
>>>>>>> >>>>>
>>>>>>> >>>>
>>>>>>> >>>
>>>>>>> >>> _______________________________________________
>>>>>>> >>> LLVM Developers mailing list
>>>>>>> >>> llvm-dev at lists.llvm.org
>>>>>>> >>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>> >>
>>>>>>> >
>>>>>>> > _______________________________________________
>>>>>>> > LLVM Developers mailing list
>>>>>>> > llvm-dev at lists.llvm.org
>>>>>>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20171122/59540d4d/attachment-0001.html>