Modular scripting in CLI

Lukas Krejci lkrejci at redhat.com
Tue Jun 5 17:19:22 UTC 2012


----- Original Message -----
> From: "John Sanda" <jsanda at redhat.com>
> To: rhq-devel at lists.fedorahosted.org
> Sent: Tuesday, June 5, 2012 6:05:53 PM
> Subject: Re: Modular scripting in CLI
> 
> Most languages provide some some mechanisms for code reuse, be it at
> the
> level of a function, classs, package, shared library, etc. JavaScript
> is
> not on of those languages. This is a major shortcoming. This is not
> an
> easy problem to solve, and fortunately as Lukas has pointed out we
> don't
> have to. There are already several existing solutions. I am in favor
> of
> moving to Rhino 1.7R3. In addition to CommonJS module support,
> JavaScript objects implement Java collection interfaces (finally) in
> this release.
> 
> As for implementing our own script engine,  I wouldn't think that it
> would be too much effort especially since Rhino already implements
> all
> of the javax.script APIs. This brings up a related point though that
> ties in with my email about doing away with built-in functions. The
> primary advantage of working through the javax.script APIs is that it
> provides a common set of APIs across script engines and languages.
>  This
> can be really nice for interopability when dealing with polygot
> programming. We however, only support JavaScript so I think it is
> worth
> asking what benefit is there of going through the extra layers and
> complexity of the javax.script APIs versus directly using Rhino. 

I think the main reason for us to go through javax.script APIs is not to close the doors from supporting other languages, too. I didn't clearly stress it before but I do believe that we should leave it for our community to decide what scripting language they'd prefer by simple "voting with action". We have our rhq-samples github project which I think would be a
great place for people to share utility scripts in whatever scripting language. By letting the community build around our support for multiple langs we effectively broaden the choice  for the community and also provide another area for external contributions. This largely overlaps with the REST "problem" you mention below.

> Two
> of
> the big features of the CLI shell are tab completion and command
> history. The Rhino REPL (like just about every other REPL for other
> languages I have used) provide those features.
> 
> It seems to me that moving to using Rhino directly is something we
> should strongly consider. The difficult problem of module support is
> already solved and we have less code to maintain.
> 

There's a lot to like about this idea, but given the point I raised above about sticking to javax.script and also because of the fact that Rhino's REPL has some feature disparity with our existing CLI (in terms of the detailed code completion we provide), I'm not sure it would be easy to try and (re)use Rhino's REPL. CommonJS support as well as enhancing the scope with RHQ's contextual variables shouldn't be too difficult but modifying the code completion as well as adding the interactive editing features we (somewhat) provide could prove complex.

> I like the idea of being able to share code between client-side and
> server-side alert CLI scripts. I think that it something worth
> further
> investigation. 

Me too :)

> Along those lines, I think we should consider
> including a
> library like underscore.js. JavaScript provides a very minimal
> standard
> library and libraries like underscore.js help fill in a lot of those
> gaps. I've used it in CLI scripts before to avoid rewriting common
> utility functions.
> 

One of the advantages of tying ourselves to CommonJS is that it defines a number of module APIs to form a sort of standard javascript library. For example Narwhal implements quite a large number of them (https://github.com/tlrobinson/narwhal/tree/master/lib).

> With respect to additional language support, I think that ought to be
> part of a related discussion on the CLI, the REST APIs, and how they
> co-exist going forward as the REST APIs mature.
> 

Yes. With REST our support for CLI becomes nearly pointless. The only reason I think it is worth sticking with our current remote APIs (and to a lesser extent with the CLI as it is or rather will be ;) ) is that the remote APIs aren't exclusively used only by CLI but rather are just a public "endpoint" that can be used by any Java-based client (and it is being used, as you know). I'll wander slightly off topic now, but I do believe that if nothing else then we must at least close the gap between remote API and REST (as far as I know they have truly diverged now with REST no longer offering only a subset of what's available in remote API).

Another reason for sticking with remote API in the CLI is the fact that we can run scripts on the serverside - currently only as alert scripts but I do hope for some other scriptable serverside functionality in the future. Using REST API directly on the server seems like a little bit of an overkill to me (and also remember the problems we had sharing the session between our GUI app and the REST war - this might or might not apply, I think there would be some differences to how reporting tried to use REST).


> On 06/04/2012 11:59 AM, Lukas Krejci wrote:
> > Hi all,
> >
> > this is to describe my recent work towards the support of modular
> > scripts in our CLI. The code I am talking about here is located in
> > a separate branch in our repo
> > http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=shortlog;h=refs/heads/lkrejci/modular-scripting.
> >
> > First of all to quickly describe what script modularity means - it
> > is the ability of a script to somehow include the code from
> > another script and use it. I.e. it is a way of splitting up your
> > scripts into reusable parts.
> >
> > I've done some preliminary investigation of possible approaches a
> > couple of months ago, captured in
> > http://wiki.rhq-project.org/display/RHQ/Script+Modularity. My main
> > point during that investigation was that we should strive for
> > bringing the support for CommonJS modules into RHQ CLI. In that
> > investigation, the only alternative I found was http://ringojs.org
> > - but that is an awful overkill, because it is a full runtime for
> > running web applications written in javascript (on the
> > serverside).
> >
> > After that, John Sanda pointed out to me that Rhino 1.7R3 itself
> > (the javascript interpreter for java) contains a CommonJS
> > compliant module loader. That means that we actually would not
> > need nothing more than Rhino 1.7R3 for gaining module support in
> > RHQ CLI. Unfortunately that is easier said than done. The version
> > of Rhino that comes bundled with the JRE is not easily swapped for
> > different version - I guess that is for a good reason. There is
> > therefore no (not awfully hacky) way of using Rhino 1.7R3 with the
> > default javascript script engine that comes with JRE.
> >
> > In my opinion having our own script engine for a specific version
> > of Rhino is by far the easiest approach for gaining the module
> > support however you slice it. The reason for this is that while
> > module loaders written in pure javascript are possible (requirejs
> > on Rhino), the implementation looks several orders of magnitude
> > more complex than what is done in Java (at least to an untrained
> > eye of a Java guy like me). The other reason is that I found no
> > pure javascript-based CommonJS-compliant module loader (requirejs
> > uses a slightly different approach called AMD) and given the
> > complexity of requirejs code, I'd really hate to write our own.
> >
> > The disadvantage of having our own script engine is obviously that
> > we then become "owners" of that script engine and have to maintain
> > it. In my humble opinion that is not a big deal actually because
> > thanks to the licensing of the Phobos project
> > https://svn.java.net/svn/scripting~svn/, we can bundle their
> > script engine (which is almost identical to the one bundled with
> > JRE, the same people wrote it actually) and provide our own little
> > additions/modifications to it. Unfortunately we cannot just extend
> > their existing impl and use it as a library but because of the way
> > the code is written and what we need to do, we have to modify the
> > source code of the script engine - in fact only a single file
> > needed update to get what we want.
> >
> > So in another words we now have a script engine
> > (http://git.fedorahosted.org/git/?p=rhq/rhq.git;a=blob;f=modules/enterprise/scripting/javascript/src/main/java/org/rhq/scripting/javascript/engine/RhinoScriptEngine.java;hb=refs/heads/lkrejci/modular-scripting)
> > that is equal (as far as I can tell) to the one bundled with JRE
> > (including the security features) and in addition makes the
> > "require" function and CommonJS support as such available out of
> > the box.
> >
> > Another very important aspect of Rhino's CommonJS implementation is
> > that the "module providers", i.e. the things that are responsible
> > for locating the source code of the module when it is being
> > loaded, are pluggable. This enables us to provide an
> > implementation that is not only able to locate a module somewhere
> > on a filesystem, but also for example in RHQ's repository. Think
> > about this code snippet:
> >
> > var my_stuff = require('repo://my-rhq-repository/my_stuff.js');
> > my_stuff.reign_the_world();
> >
> > Sharing common modules between remote CLI scripts and server-side
> > alert notifications? Putting all the utility code in a public
> > repository on the server and not having to download them to each
> > CLI installation? Prepackaging RHQ samples and/or utility code on
> > the serverside so that they can be used by alert scripts, too?
> > (I'd imagine this being handled by yet another "scheme" like
> > "require('rhq://util');" being equivalent to today's "exec -f
> > 'samples/util.js';" with the exception that the actual code would
> > be stored on the serverside so that both CLI scripts and alert
> > scripts could use it).
> >
> > But that's not all that has been done. You may remember from the
> > last year's F2F meeting that I put together a crude version of
> > python-backed CLI just during John Sanda's presentation on another
> > topic (and I apologize to John once again for being so rude - it
> > WAS an interesting presentation ;) ). I was able to do that that
> > quickly because our current code is 99% language independent. I
> > thought that if we are going the way of custom script engines and
> > bundling language interpreters it would be really a shame not to
> > add that 1% and make the CLI code completely language independent
> > and push the language support into standalone "modules". Why do I
> > think it is useful? Well, speaking of John Sanda again, he likes
> > Clojure, admins, I imagine, would prefer python over javascript,
> > script kiddies would probably prefer ruby over python and someone
> > else might like groovy. Do we need to support them all in JBoss
> > ON? I'd say no, but at the same time we should not artificially
> > limit our users - be it RHQ or JBoss ON users - from using the
> > scripting language of their choice if the work required on our
> > side amounts to adding 1% to the 99% complete puzzle.
> >
> > So that's almost done, too... There are a couple of new modules:
> > modules/enterprise/scripting/api defining the API that needs to be
> > implemented by new language support modules and
> > modules/enterprise/scripting/javascript and even
> > modules/enterprise/scripting/python (which is not complete yet).
> > The language support modules are discovered using
> > META-INF/services mechanism and are made available for
> > instantiation through RHQ's ScriptEngineFactory (which required no
> > API changes for this, remember we were 99% there). The thing that
> > is missing is the updates needed to our build process for bundling
> > the javascript language support with the CLI (maybe we could later
> > think about having other language support modules being
> > downloadable from the server from Administration/Downloads?) and
> > porting the code completion code from the CLI itself into the
> > javascript language module (which should be fairly easy as far as
> > I checked). (Other languages would probably lack the code
> > completion altogether but I think we should think about providing
> > some interactive API help rather than trying to produce a perfect
> > REPL with code completion - and that could work across different
> > languages more easily.)
> >
> > So there you go. Bringing modularity support to javascript wasn't
> > that hard after all and as a bonus we might gain the ability to
> > support other scripting languages, too ;) Unfortunately there is
> > nothing to try out yet (you can only stare at the code), because
> > as I wrote above, I didn't connect all the dots yet (even though
> > I'm close). I wanted to give you this overview of what's been done
> > and what it'll bring us so that we can discuss the approach I took
> > and hopefully make some enhancements to it.
> >
> > Cheers,
> >
> > Lukas
> >
> >
> >
> >
> > _______________________________________________
> > rhq-devel mailing list
> > rhq-devel at lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/rhq-devel
> 
> 
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel
> 


More information about the rhq-devel mailing list