[lldb-dev] bindings as service idea

[Zachary Turner via lldb-dev]

On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner <zturner at google.com> wrote:

> Well some of the bugfixes are actually worth mentioning, because we
> actually have bugs on the C++ side that we can't fix because then SWIG
> won't be able to process the header files.  For example, if SWIG sees this
> in a header file, it errors out and can't even proceed.
>
> enum Foo : unsigned {
>     Bar = 0xFFFFFFFF
> };
>
> So instead we have to write this:
>
> enum Foo {
>     Bar = 0xFFFFFFFF
> };
>
> According to the standard this produces an unsigned enum, but MSVC is
> non-comformant here, and we get a signed enum.  So we have hundreds of
> warnings disabled as a result of this, and it's a legitimate bug on the C++
> side that we would like to fix, irrespective of SWIG.
>
> Feature-wise, here's a potentially incomplete list:
>
> * Typemaps can re-use other typemaps, similar to how functions can call
> other functions
> * A new -debug-tmsearch command line option which helps debugging typemap
> problems.  For example, when you're looking at some generated code and
> trying to figure out which typemap generated it.  I know I've had to do
> this many times during the Python 3 conversion, and this would have helped.
> * Template classes are supported.  We probably don't care about this, but
> it's an interesting thought.
> * -builtin option supports generating higher performance wrapping code.
> Read the SWIG documentation about -builtin
>
Grr, do embedded links not work on the mailing list or something?

http://www.swig.org/Doc3.0/Python.html#Python_builtin_types


>
> JavaScript support has been mentioned as someone someone needs on more
> than one occasion.  The name of the person interested escapes me, but this
> wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x
> of SWIG either, I don't believe.
>
> I'll try to think of some more.
>
>
> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala <todd.fiala at gmail.com> wrote:
>
>> I'm out next week, but I can help if needed after that.
>>
>> Related to all this, you have mentioned a few times that there are newer
>> swig features you want to use.
>>
>> Can you enumerate the features not present in 1.x but present in 3.x that
>> you want to take advantage of, and what benefits they will bring us?  (I'm
>> not referring to bug fixes in bindings, but actual features that bring
>> something new that we didn't have before).
>>
>> Thanks!
>>
>> -Todd
>>
>> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner <zturner at google.com>
>> wrote:
>>
>>> I wasn't planning on working on this immediately, but given the outcome
>>> of the recent static bindings work, I can re-prioritize.  I don't know how
>>> long it will take, because honestly writing this kind of thing in Python is
>>> new to me.. to make an understatement.  But I'll get it done.  Give me
>>> until mid next week and I'll post an update.
>>>
>>> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala <todd.fiala at gmail.com>
>>> wrote:
>>>
>>>> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner <zturner at google.com>
>>>> wrote:
>>>>
>>>>> Just to re-iterate, if we use the bindings as a service, then I
>>>>> envision checking the bindings in.  This addresses a lot of the potential
>>>>> pitfalls you point out, such as the "oops, you can't hit the network, no
>>>>> build for you" and the issue of production build flows not wanting to hit a
>>>>> third party server, etc.
>>>>>
>>>>> So if we do that, then I don't think falling back to local generation
>>>>> will be an issue (or important) in practice.  i.e. it won't matter if you
>>>>> can't hit the network.  The reason I say this is that if you can't hit the
>>>>> network you can't check in code either.  So, sure, there might be a short
>>>>> window where you can't do a local build , but that would only affect you if
>>>>> you were actively modifying a swig interface file AND you were actively
>>>>> without a network connection.  The service claims 99.95% uptime, and it's
>>>>> safe to say we are looking at significantly less than 100% usage of the
>>>>> server (given checked in bindings), so I think we're looking at once a year
>>>>> -- if that -- that anyone anywhere has an issue with being able to access
>>>>> the service.
>>>>>
>>>>>
>>>> That seems fine.
>>>>
>>>>
>>>>> And, as you said, the option can be provided to change the host that
>>>>> the service runs on, so someone could run one internally.
>>>>>
>>>>> But do note, that if the goal here is to get the SWIG version bumped
>>>>> in the upstream, then we will probably take advantage of some of these new
>>>>> SWIG features, which may not work in earlier versions of SWIG.  So you
>>>>> should consider how useful it will be to be able to run this server
>>>>> internally, because if you can't run a new version of swig locally, then
>>>>> can you run it internally anywhere?  I don't know, I'll leave that for you
>>>>> to figure out.
>>>>>
>>>>>
>>>> That also seems fine.  And yes, we can work it out on our end.
>>>>
>>>> We'd need to make sure that developer flows would pick up the need to
>>>> generate the bindings again if binding surface area changed, but that is no
>>>> different than now.
>>>>
>>>>
>>>>> Either way, it will definitely have the ability to use a different
>>>>> host, because that's the easiest way to debug theclient and server (i.e.
>>>>> run them on the same machine with 127.0.0.1)
>>>>>
>>>>>
>>>> Yep, sounds right.
>>>>
>>>>
>>>>> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala <todd.fiala at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> For the benefit of continuity in conversation, here is what you had
>>>>>> to say about it before:
>>>>>>
>>>>>> > One possibility (which I mentioned to you offline, but I'll put it here for
>>>>>> others to see) is that we make a swig bot which is hosted in the cloud much
>>>>>> like our public build bots.  We provide a Python script that can be run on
>>>>>> your machine, which sends requests over to the swig bot to run swig and
>>>>>> send back the results.  Availability of the service would be governed by
>>>>>> the SLA of Google Compute Engine, viewable here:https://cloud.google.com/compute/sla?hl=en
>>>>>>
>>>>>> > If we do something like this, it would allow us to raise the SWIG version
>>>>>> in the upstream, and at that point I can see some benefit in checking the
>>>>>> bindings in.  Short of that, I still dont' see the value proposition in
>>>>>> checking bindings in to the repo.  [bits deleted]
>>>>>>
>>>>>> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
>>>>>> to make remote swig generation service and get it up and running.
>>>>>>
>>>>>>
>>>>>> I'd like feedback from others on this.  Is this something we want to consider doing?
>>>>>>
>>>>>> From my perspective, this seems reasonable to look into doing if we:
>>>>>>
>>>>>> (a) have the service code available, and
>>>>>>
>>>>>> (b) if we so choose, we can readily have the script hit another server (so that a consumer can have the entire setup on an internal network), and
>>>>>>
>>>>>> (c: option 1) be able to fall back to generate with swig locally as we do now in the event that we can't hit the server
>>>>>>
>>>>>> (c: option 2) rather than fall back to swig generation, use swig generation as primary (as it is now) but, if a swig is not found, then do the get-bindings-as-a-service flow.
>>>>>>
>>>>>> This does open up multiple ways to do something, but I think we need to avoid a failure mode that says "Oops, you can't hit the network.  Sorry, no lldb build for you."
>>>>>>
>>>>>>
>>>>>> Reasoning:
>>>>>>
>>>>>> For (a): just so we all know what we're using.
>>>>>>
>>>>>> For (b): I can envision production build flows that will not want to be hitting a third-party server.  We shouldn't require that.
>>>>>>
>>>>>> For (c): we don't want to prevent building in scenarios that can't hit a network during the build.
>>>>>>
>>>>>>
>>>>>> -Todd
>>>>>>
>>>>>>
>>>>>> On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala <todd.fiala at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hey Zachary,
>>>>>>>>
>>>>>>>> I think the time pressure has gotten the better of me, so I want to
>>>>>>>> apologize for getting snippy about the static bindings of late.  I am
>>>>>>>> confident we will get to a good solution for removing that dependency, but
>>>>>>>> I can certainly wait for a solution (using an alternate approach in our
>>>>>>>> branch) until we arrive at something more palatable to everyone.
>>>>>>>>
>>>>>>>> Regarding the bindings as service idea:
>>>>>>>>
>>>>>>>> How quickly do you think you could flesh out the bindings as a
>>>>>>>> service idea?  With a relatively strong dislike for the static approach I'm
>>>>>>>> taking, I can back off that and just use my current code here in a
>>>>>>>> downstream branch for now.  Ultimately I want to remove the requirement for
>>>>>>>> swig, but I can probably achieve that without doing it in upstream if we're
>>>>>>>> going to have some solution there at some point ideally sooner than later.
>>>>>>>>
>>>>>>>> Also - I think you were going to send me a swig 3.x binding to try
>>>>>>>> out (I'd need the LLDBWrapPythoh.cpp and the lldb.py, and you'd just need
>>>>>>>> to let me know if it still needs to be post-processed or it would need to
>>>>>>>> be done).  Can we shoot for trying that out maybe tomorrow?
>>>>>>>>
>>>>>>>>
>>>>>>> Hey I got these, Zachary.  They just didn't go in my inbox.
>>>>>>>
>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> --
>>>>>>>> -Todd
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> -Todd
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> -Todd
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> -Todd
>>>>
>>>
>>
>>
>> --
>> -Todd
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151119/0192bed7/attachment-0001.html>