On 11/23/2011 12:32 PM, Zane Bitter wrote:
Interesting. Russell and I were discussing this earlier and were
sort-of coming to the
conclusion that as far as possible we want to port Gofer to Matahari (i.e. have existing
plugins and clients continue to work) rather than just raiding it for code. But what
you're saying seems to suggest going the other way.
Not sure. Just thinking that if the objective is to support users writing matahari agents
in python, users will want to access those agent(s) using the same matahari client as they
would for C++ matahari agents. That said, I see any integration as being something in the
matahari project that uses gofer as the upstream.
Along these lines, I can't help but wonder if the goal is to provide a means for
matahari
users to write agents in python, maybe the value of an integration with gofer really
isn't
that high. After replacing the messaging layer with QMF, what's left is a plugin and
dispatching model. Given that C++ matahari agents don't follow a plugin model, it
seems
odd to introduce one for python. In fact, it seems contradictory to the each agent being
a single API (an API per daemon). Although I completely understand the desire to not
reinvent the wheel, we should consider how much of the existing (gofer) wheel matahari
wants. Dispatching function calls from C++ into python is fairly straight forward.
Perhaps this would be simpler?:
* Adapt the matahari client to be able to marshal arguments/returned values
using JSON as you noted.
* Provide for a python module to be specified in the agent configuration as needed.
* When a python module is specified, the agent loads the module and dispatches API
calls to functions in the python module.
* Migrating an existing gofer plugin to a matahari agent would be a simple
cut-n-paste exercise.
I think this could be up and running in a few days.
Just thinking out loud :)
On 23/11/11 18:19, Jeff Ortel wrote:
> On 11/17/2011 11:53 AM, Zane Bitter wrote:
>> Dynamically generating a schema is no problem, but there is no way to
>> know the correct
>> types for method arguments. In fact, there is no way in general to
>> even know the correct
>> *number* of arguments. On reflection, I think the way to do this is to
>> have a single QMF
>> argument to each method called "args" (actually two, one for named
>> args as well). We could
>> pass this argument simply as a string in JSON as Gofer is doing now,
>> or split it into an
>> FTABLE (QMF's dictionary/hash type) leaving the individual arguments
>> in JSON format. So
>> all Gofer methods would effectively have identical schemata.
>
> I think that regardless of whether arguments are passed into the python
> layer as the correct types or a as a JSON blob, we still need to account
> for argument types. Although python uses late binding, methods may
> assume that arguments are typed as (at least), str|int|float. So, even
> if the matahari plugin mimics the gofer JSON envelpoe, the contents will
> still need to be typed (from a JSON perspective).
This is true, although we can't really talk about it independently of the question of
what
client libraries will be available.
Agreed.
I think we can pretty easily provide a special case
for Gofer in the Matahari Python API to convert all arguments to JSON
strings (in most
cases probably just by calling repr() on them), and the shell could inherit this special
case as well. If we were to port the existing Gofer client library, we would do the same
(which is presumably also what it does now).
Basically. Gofer builds a dict representing the (messaging) envelope which is handed to
simple-json.
This seams like a reasonable approach for marshalling the parameters and presumably the
returned values and propagated exceptions.
How would the matahari client know it needs to marshall parameters/returned values in
JSON?
Anyone trying to use the QMF API directly rather than going through the Matahari API
would
be on their own (the schema would just specify a list of strings; they would have to
convert the arguments to JSON themselves. I'm not sure that this is a great hardship,
and
on Python at least I don't see any reason why people would use the generic QMF API
once we
have the Matahari one out. (This does make the issue of Ruby support more pressing
though.
However Ruby is not likely to make it into the first release anyway.)
> What if the gofer plugin container could be inspected for class and
> functions where each function or class method would provide a signature?
> The signature would include the method name and argument names and types.
I'm not a fan of adding this kind of complexity (effectively type annotations) for
plugin
authors. If you call a local function in Python that was expecting arguments of a
different type, you get an exception. I don't think an RPC implementation need be
*more*
robust than that.
Agreed.
I was thinking this could be done without burden to plugin writers - but it can't. I
definitely would not want to introduce type annotations or anything like that. As you
said, python (gofer) relies on the caller to pass the correct args (and arg types) and
JSON encoding/decoding to handle this.
Yes, we're ignoring a perfectly good system for conveying type information to clients
and
forcing people to rely on documentation. But in these circumstances it seems like a
reasonable trade-off.
>> This is not exactly pretty, but it is in keeping with the dynamic
>> nature of the target
>> languages. We could special-case the Gofer agent in the Matahari shell
>> to make accessing
>> it through mhsh a little tidier.
>>
>> This brings me to another question: would we port the client-side
>> Gofer libraries to use
>> the Matahari/QMF backend? I think that would be relatively easy, and
>> certainly this format
>> could be implemented without users accessing it through that API ever
>> noticing.
>
> No. The gofer client provides for a features that I'm 99% sure that qmf
> does not support. Primarily in the areas of, object constructors,
> authentication/authorization and timeouts (perhaps others). Because of
> the differences in this functionality end-to-end, I anticipated that one
> restriction of accessing gofer plugins with the matahari/qmf environment
> is that user use the qmf client(s) and write agent plugins accordingly.
I'm not sure that QMF presents a big obstacle to any of those. For timeouts, QMF
supports
asynchronous operation. Some of the stuff like where there is a timeout specified for
both
starting and completing an event would be more complicated - we could either leave that
out or implement the agent asynchronously and have it fire back an immediate method
response followed later by an event on completion of the call.
Authentication/authorisation is presumably just more parameters to pass. I'm not sure
what
you're getting at with the object constructors?
Perhaps you're right. If we can pass whatever data we need through the QMF layer, we
should be able to support these things.
By supporting arguments passed to remote class constructors, plugin writers can write more
natural remote objects. It's a semantic convenience.
Again, this stuff might not look pretty to someone trying to access it through the
generic
QMF API, but I'm not aware of any reason that we couldn't implement it in our
API. Are
there any particular missing features in QMF that you could point to?
thanks,
Zane.