[lldb-dev] bindings as service idea

Thu Nov 19 10:57:13 PST 2015

I don’t think so, this was just an embedded link to your hard drive:

file:///C:/tools/swigwin-3.0.7/Doc/Manual/Python.html#Python_builtin_types

Sean

> On Nov 19, 2015, at 10:51 AM, Zachary Turner via lldb-dev <lldb-dev at lists.llvm.org> wrote:
> 
> 
> 
> On Thu, Nov 19, 2015 at 10:50 AM Zachary Turner <zturner at google.com <mailto:zturner at google.com>> wrote:
> Well some of the bugfixes are actually worth mentioning, because we actually have bugs on the C++ side that we can't fix because then SWIG won't be able to process the header files.  For example, if SWIG sees this in a header file, it errors out and can't even proceed.
> 
> enum Foo : unsigned {
>     Bar = 0xFFFFFFFF
> };
> 
> So instead we have to write this:
> 
> enum Foo {
>     Bar = 0xFFFFFFFF
> };
> 
> According to the standard this produces an unsigned enum, but MSVC is non-comformant here, and we get a signed enum.  So we have hundreds of warnings disabled as a result of this, and it's a legitimate bug on the C++ side that we would like to fix, irrespective of SWIG.
> 
> Feature-wise, here's a potentially incomplete list:
> 
> * Typemaps can re-use other typemaps, similar to how functions can call other functions
> * A new -debug-tmsearch command line option which helps debugging typemap problems.  For example, when you're looking at some generated code and trying to figure out which typemap generated it.  I know I've had to do this many times during the Python 3 conversion, and this would have helped.
> * Template classes are supported.  We probably don't care about this, but it's an interesting thought.
> * -builtin option supports generating higher performance wrapping code.  Read the SWIG documentation about -builtin <>
> Grr, do embedded links not work on the mailing list or something?
> 
> http://www.swig.org/Doc3.0/Python.html#Python_builtin_types <http://www.swig.org/Doc3.0/Python.html#Python_builtin_types>
>  
> 
> JavaScript support has been mentioned as someone someone needs on more than one occasion.  The name of the person interested escapes me, but this wasn't added until SWIG 3.x.  Other languages like Go and Lua aren't in 1.x of SWIG either, I don't believe.
> 
> I'll try to think of some more.
> 
> 
> On Thu, Nov 19, 2015 at 10:17 AM Todd Fiala <todd.fiala at gmail.com <mailto:todd.fiala at gmail.com>> wrote:
> I'm out next week, but I can help if needed after that.
> 
> Related to all this, you have mentioned a few times that there are newer swig features you want to use.
> 
> Can you enumerate the features not present in 1.x but present in 3.x that you want to take advantage of, and what benefits they will bring us?  (I'm not referring to bug fixes in bindings, but actual features that bring something new that we didn't have before).
> 
> Thanks!
> 
> -Todd
> 
> On Thu, Nov 19, 2015 at 10:14 AM, Zachary Turner <zturner at google.com <mailto:zturner at google.com>> wrote:
> I wasn't planning on working on this immediately, but given the outcome of the recent static bindings work, I can re-prioritize.  I don't know how long it will take, because honestly writing this kind of thing in Python is new to me.. to make an understatement.  But I'll get it done.  Give me until mid next week and I'll post an update.
> 
> On Thu, Nov 19, 2015 at 10:12 AM Todd Fiala <todd.fiala at gmail.com <mailto:todd.fiala at gmail.com>> wrote:
> On Thu, Nov 19, 2015 at 9:44 AM, Zachary Turner <zturner at google.com <mailto:zturner at google.com>> wrote:
> Just to re-iterate, if we use the bindings as a service, then I envision checking the bindings in.  This addresses a lot of the potential pitfalls you point out, such as the "oops, you can't hit the network, no build for you" and the issue of production build flows not wanting to hit a third party server, etc.  
> 
> So if we do that, then I don't think falling back to local generation will be an issue (or important) in practice.  i.e. it won't matter if you can't hit the network.  The reason I say this is that if you can't hit the network you can't check in code either.  So, sure, there might be a short window where you can't do a local build , but that would only affect you if you were actively modifying a swig interface file AND you were actively without a network connection.  The service claims 99.95% uptime, and it's safe to say we are looking at significantly less than 100% usage of the server (given checked in bindings), so I think we're looking at once a year -- if that -- that anyone anywhere has an issue with being able to access the service.
> 
> 
> That seems fine.
>  
> And, as you said, the option can be provided to change the host that the service runs on, so someone could run one internally.
> 
> But do note, that if the goal here is to get the SWIG version bumped in the upstream, then we will probably take advantage of some of these new SWIG features, which may not work in earlier versions of SWIG.  So you should consider how useful it will be to be able to run this server internally, because if you can't run a new version of swig locally, then can you run it internally anywhere?  I don't know, I'll leave that for you to figure out.
> 
> 
> That also seems fine.  And yes, we can work it out on our end.
> 
> We'd need to make sure that developer flows would pick up the need to generate the bindings again if binding surface area changed, but that is no different than now.
>  
> Either way, it will definitely have the ability to use a different host, because that's the easiest way to debug theclient and server (i.e. run them on the same machine with 127.0.0.1)
> 
> 
> Yep, sounds right.
>  
> On Thu, Nov 19, 2015 at 8:00 AM Todd Fiala <todd.fiala at gmail.com <mailto:todd.fiala at gmail.com>> wrote:
> For the benefit of continuity in conversation, here is what you had to say about it before:
> 
> > One possibility (which I mentioned to you offline, but I'll put it here for
> others to see) is that we make a swig bot which is hosted in the cloud much
> like our public build bots.  We provide a Python script that can be run on
> your machine, which sends requests over to the swig bot to run swig and
> send back the results.  Availability of the service would be governed by
> the SLA of Google Compute Engine, viewable here:
> https://cloud.google.com/compute/sla?hl=en <https://cloud.google.com/compute/sla?hl=en>
> 
> > If we do something like this, it would allow us to raise the SWIG version
> in the upstream, and at that point I can see some benefit in checking the
> bindings in.  Short of that, I still dont' see the value proposition in
> checking bindings in to the repo.  [bits deleted]
> 
> > If it means we can get off of SWIG 1.x in the upstream, I will do the work
> to make remote swig generation service and get it up and running.
> 
> I'd like feedback from others on this.  Is this something we want to consider doing?
> From my perspective, this seems reasonable to look into doing if we:
> (a) have the service code available, and
> (b) if we so choose, we can readily have the script hit another server (so that a consumer can have the entire setup on an internal network), and
> (c: option 1) be able to fall back to generate with swig locally as we do now in the event that we can't hit the server
> (c: option 2) rather than fall back to swig generation, use swig generation as primary (as it is now) but, if a swig is not found, then do the get-bindings-as-a-service flow.
> This does open up multiple ways to do something, but I think we need to avoid a failure mode that says "Oops, you can't hit the network.  Sorry, no lldb build for you."
> 
> Reasoning:
> For (a): just so we all know what we're using.
> For (b): I can envision production build flows that will not want to be hitting a third-party server.  We shouldn't require that. 
> For (c): we don't want to prevent building in scenarios that can't hit a network during the build.
> 
> -Todd
> 
> On Wed, Nov 18, 2015 at 10:58 PM, Todd Fiala <todd.fiala at gmail.com <mailto:todd.fiala at gmail.com>> wrote:
> 
> 
> On Wed, Nov 18, 2015 at 10:06 PM, Todd Fiala <todd.fiala at gmail.com <mailto:todd.fiala at gmail.com>> wrote:
> Hey Zachary,
> 
> I think the time pressure has gotten the better of me, so I want to apologize for getting snippy about the static bindings of late.  I am confident we will get to a good solution for removing that dependency, but I can certainly wait for a solution (using an alternate approach in our branch) until we arrive at something more palatable to everyone.
> 
> Regarding the bindings as service idea:
> 
> How quickly do you think you could flesh out the bindings as a service idea?  With a relatively strong dislike for the static approach I'm taking, I can back off that and just use my current code here in a downstream branch for now.  Ultimately I want to remove the requirement for swig, but I can probably achieve that without doing it in upstream if we're going to have some solution there at some point ideally sooner than later.
> 
> Also - I think you were going to send me a swig 3.x binding to try out (I'd need the LLDBWrapPythoh.cpp and the lldb.py, and you'd just need to let me know if it still needs to be post-processed or it would need to be done).  Can we shoot for trying that out maybe tomorrow?
> 
> 
> Hey I got these, Zachary.  They just didn't go in my inbox.
>  
> Thanks!
> -- 
> -Todd
> 
> 
> 
> -- 
> -Todd
> 
> 
> 
> -- 
> -Todd
> 
> 
> 
> -- 
> -Todd
> 
> 
> 
> -- 
> -Todd
> _______________________________________________
> lldb-dev mailing list
> lldb-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/lldb-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/lldb-dev/attachments/20151119/e3a023ff/attachment-0001.html>