Notifications between Aeolus components

Martin Povolny mpovolny at redhat.com
Thu Jan 24 23:18:46 UTC 2013


Matt, please, do not take anything I write personally or as an offense.
I started with saying that all these as IMHOs. Most my ideas are based
on my previous experience from outside this project and although I think
that 4 months is enough time to get the idea of Conductor, I surely
don't have much idea about the history of the project, don't know all
the components in much detail and cannot yet understand reasons that led
to previous decisions that might seem strange to me.

I will try to address the points you made one by one, no offense.

On Thu, Jan 24, 2013 at 01:55:09PM -0500, Matt Wagner wrote:
> On Thu, Jan 24, 2013 at 10:10:47AM +0100, Martin Povolny wrote:
> > 
> > I was not here when this decisions where made and did not hear any
> > reasonable argument why the MQ was removed from the project.
> 
> I'm trying to find the mailing list thread, but I think it predates the
> aeolus-devel.
> 
> But suffice it to say that a lot of smart people thought about it and
> the community decided it was the right path forward at the time. That's
> not to say that it means we shouldn't consider re-adding it, just that
> your not seeing "any reasonable argument" doesn't mean that the decision
> at the time was wrong.

Sorry, that is not an argument. It also does not mean that the decision
was right.

> 
> > So just my IMHO
> > 
> > Conductor has 2 components that are sort of "daemons"
> > 
> > * dbomatic
> > * delayed job
> > 
> > Then there is the web ui part.
> > 
> > The communication layer for the 3 is the dababase.
> 
> I wouldn't characterize it as "communication" through the database,
> though it might be technically correct.

I would. That is exactly what the components are doing. One writes
something in the DB the other reads it and acts upon it, then it writes
something back for the first one. That is communication.

> 
> dbomatic is a script that polls Deltacloud and updates the database. It
> predates our use of delayed_job. I think dbomatic is universally
> accepted as something that needs major overhaul at this point, and does
> a lot of things in weird ways. But the whole concept is doing what you
> seem to propose later in this message -- moving the long-running stuff
> away from the core UI/API code.
> 
> Similarly, delayed_job runs jobs enqueued in the database. It's widely
> used in the Rails community, so it's not as if it's unholy thing writing
> to the database like dbomatic started out as. It's true that it reads
> things written to the database by Conductor, so in that sense it's
> 'communicating' through the database, but I see nothing wrong with this
> approach, and presumably all of the other people using and contributing
> to delayed_job would agree.

Sorry, but a few IMHOs here. Dbomatic cannot even handle database
restart. When I first saw it, I thought "Seriously? Is it possible? No
Way! WAT!"

And the argument that delayed_job is widely used in the Rails community
is not valid for me.

It justs gives me a hope that it might handle database disconnects
better then the dbomatic. Does it?

> 
> > IMHO it would make much more sense to have a "backend conductor" that
> > would do the stuff that really matters including servicing dome sort of
> > job-queue (now dbomatic) and
> >   >> communicating with the other components <<
> > including polling where necessary (now delayed job).
> > 
> > Then the web part would be a thin layer to "display the results" -- the
> > web ui part.
> > 
> > As opposed to current situation where we have conductor the web part
> > that:
> > 
> > 1) serves the web users
> > 2) serves the API
> 
> This was a separate thread, but this is a common pattern in Rails.
> Controllers can service web and API requests from the same methods,
> reducing code duplication or inadvertent divergence between API and web
> functionality.

Sorry I don't play this game. I was taught to think critically and it
someone does something I don't need to do the same.

The present state of the controllers is a mess and splitting
responsibilities is a way out of it. Surely you need more then just a
model and a controller if you have complicated logic.

> 
> > 3) accepts the "RESTful" callbacks from ImageFactory
> 
> Isn't accepting callbacks just an extension of the API?

Sure it is. But again: The complexity. Then the error handling. Then the
Single responsibility principle.

> 
> > 4) also does some communication with DC and maybe other parts while
> > answering a client's request
> 
> I don't see how we could separate this out -- it's often integral to
> servicing the user's requests (web or API). Where it can happen in the
> background, we should certainly be throwing those tasks into
> delayed_job, but I think we're already doing that today.

Yes, more tasks are being moved to delayed job, it helps but it is not
enough.

> 
> > IMHO this is a mess from the architecture point of view.
> 
> I disagree. There's room for improvement, for sure, but I don't think
> breaking Conductor apart in ways that aren't commonly practiced in the
> Rails community is going to do anything but make the thing more of a
> confusing mess.

Yes we do not share the same view here at all.

> 
> > From my point of view it is much more important to solve this mess and
> > have the communication
> >     >> happen in the right place <<
> > than deciding whether the communication is HTTP based (REST, Message,
> > RPC, whatever) or if we use MQ.
> 
> I think I agree with this.
> 
> > The question of MQ of then comes secondary.
> > 
> > I don't know why you guys failed with the MQ in the past but I see the
> > MQs today at a similar level as SQL databases. It's an industrial
> > verified way of doing communication in situations such as ours.
> 
> I agree that it's a stable and successful way of doing things. I'm just
> not convinced it's the right choice for us.
> 
> > We cannot directly link because:
> > 
> > a) we use a REST proxy for REST provider APIs (the deltacloud)
> > although it's written in the same language as Conductor and it's
> > stateless, it's not a library and it's to stay that way I understood.
> 
> I think the library thing might be a point of minor controversy. I know
> some have expressed interest in using it this way, but it sounds like
> it's not something Deltacloud is interested in implementing.

Yes. I understood respected deltacloud people do not want to have a
library. And they have a successful project, their word has it's weight.

But is does not change my opinion that it would be better to have a
library to translate one REST call into another rather then having a
REST service to do the REST call for me.

> 
> In any case, though, they present a REST API which we use with great
> results. I believe we could mount it as a Rack app if we wanted, but I'm
> not sure that would really help us any.

Yes, that idea came to me too. Why don't we mount it? But I am not such
Rails expert to know it one can mount a Rack app into a Rails app and if
it would be beneficial i.e. we could then call the API directly.

> 
> > b) we use components written in other languages (the ImageFactory)
> 
> Sure. This can easily be solved using an HTTP API (as we do today) or
> AMQP for exchanging data. We previously tried AMQP and fairly recently
> switched to HTTP.

Well I am not sure about the "easy". I am afraid the devil is in the
detail and looking at the hooks and e.g. at the credentials being passed
I thing the problem is more complex than it might seem at the first
sight.

> 
> > c) we might want to support a scenario where the individual components
> > run on different machines.
> 
> Yes, and I worry my previous reply may have come across as dismissing
> this. This is definitely a possibility, and something we should support.
> 
> > We have more then a pair of communicating parties.
> > 
> > We have more the one party communicating with other parties.
> 
> And indeed, AMQP makes this problem a little bit easier, though I'll
> note that we've already got something in place here, so it'd be
> reimplementing something that we've already solved for the sake of
> swapping in something slightly cleaner for this use case.
> 
> > We have or want to have optional components.
> 
> Interestingly, this is one of my arguments _against_ using AMQP. HTTP is
> sort of a 'lowest common denominator' that's pretty easy to implement or
> inteface with. For purpose-built applications that have clearly-defined
> ways of connecting, AMQP is probably a big win. But for products that
> can work together or be used independently, I'm not sure everyone would
> want to write an AMQP interface to their application.

I see this argument in various contexts, let me present a twisted one:

We have a Rails engine that allows a Rails app to use a python written
ImageFactory. What is the common denominator here?

The Rails I guess. We hope other Rails projects will be able to use the
TIM. That's a small niche, isn't it?

Here you argue that the AMQP might be limiting the usage of the
components we create if we use it for internal communication.

How big is the Rails app wanting to use ImageFactory niche compared to
the niche of tools that can and want to connect to an AMQP service?

Do we create the components to be easily used by other projects?

Or do we create a components that fit best our internal needs?

Where stands TIM and where stands the potential usage of AMQP?

> 
> > There are reasons for using MQ over ad-hoc communication between
> > components and I would thing that it's not necessary to write that, but:
> > 
> > * as "Steve Loranz" pointed out: there's just one point to communicate with
> > in case of MQ
> 
> This is a bit easier, admittedly, but I haven't seen the current system
> as being too complicated. I think the most complicated is Conductor,
> which has to talk to Imagefactory and Deltacloud. (And I'm not sure that
> Deltacloud will ever use AMQP.)

You have to know better then me.

I don't have yet a good idea about how hard it can be to persuade other
project to change things in such way that would be beneficial to other
project funded by the same company. But I have a feeling that it might
be difficult ;-)

> 
> > * reliability -- not easy to get this with ad-hoc solution
> 
> I'll grant that AMQP would be more reliable. My question, really, is:
> are we having such a reliability problem that it's worth rewriting the
> entire thing?

I don't thing it would be necessary to rewrite the whole thing if we
used AMQP.

> 
> > * then of course you have the bindings for all the necessary language
> 
> I think both languages are equal here. Is there any language out there
> that has AMQP bindings but no support for HTTP? Either way, it's a moot
> point for us: Factory (Python) and Conductor (Ruby) have both used AMQP
> previously and now use REST calls, so both will clearly work either way.
> I imagine the same could be said if we ever implement C or Java or
> whatever apps -- either way will work.

The difference is the level of services provided. I hope we do not need
to argue about the fact that HTTP is much lower lever service and much
more has to be done around a HTTP request than around a AMQP request.

This brings up the error handling as an example again.

> 
> > * that someone else is using and testing for you, you have known and
> >   documented ways of "doing it" for various types of message exchange
> >   scenarios
> 
> Well the use and testing occurs at the protocol level, versus our
> implementation. I could make the same claim of HTTP.

The same argument as above applies here.

> 
> AMQP likely does have best practices around this that are more advanced
> than with webhooks, though there are plenty of projects (GitHub,
> WordPress, etc.) using them successfully.
> 
> But I don't think we're having a problem right now with figuring out how
> to implement webhooks -- they're already there -- so I'm not sure this
> one matters to us.

They are there but how much work has to be done around them to from now
on so that we can say: "It is done"?

Was it yesterday then you patched the ugly errors that where in the
Conductor log on IMF callbacks?

> 
> > * and then you have the MQ implemented, debugged, tested and working
> 
> We already have this more or less for our REST-based system, do we not?
> Switching would _require_ implementation, debugging, and testing. It's
> surely doable, but I don't think this is a reason to switch.

It's not a reason to which. It's just one more argument for using AMQP.
You said there where no reasons or at least I understood so.

> 
> > * message format, error handling etc.
> > 
> >   see the problems we have when trying to handle the various error
> >   conditions -- how many times do you get a reasonable error message in
> >   conductor when IMF or DC fails? this is also easier to do with a MQ
> 
> We do a bad job with error-handling, it's true. It's especially bad
> between components. But the problem is that we just don't do a good job
> of spelling out what the possible errors are or how we should present
> them. This isn't going to change with a message bus -- it's going to
> change when we actually fix our handling of errors.

I don't agree here. I find the error handling generally easier with a
higher level tool. And it applies when comparing HTTP and AMQP for sure.

> 
> > * we can go on -- google knows
> 
> But this is exactly my point here -- I don't think the overall
> advantages of AMQP matter here. What matters is how it will make _our
> project_ better.

I think that that was shown.

> 
> > To get job done, its good to write less code. It might not be the
> > scientific approach but it works (IMHO).
> 
> I agree with that, but I don't think switching our communication
> protocols will help this.

We have different opinions here too.

> 
> > Somewhere in the thread someone said that it seems we decided to use MQ
> > because it is "cool" and seek the reasons why to do so. I don't see it
> > this way.
> > 
> > What I see much more the decision NOT to use the MQ and ignoring the
> > good reasons to use one.
> 
> There's admittedly some lingering animosity towards AMQP here from the
> last time we used it -- we put a bunch of effort into making it work,
> had all sorts of problems with it, and then decided to rip it out and
> reimplement REST callbacks. Those that were involved are probably going
> to be pretty reluctant to now rip out the REST bits and implement AMQP.
> 
> What I'm interested in aren't the "good reasons" that AMQP is a superior
> messaging protocol, but the reasons it will make _our project_ better.
> The only one I've heard so far is that message delivery will be more
> robust. That would be an improvement but I'm not sure it's worth the
> overhead of implementing this. Maybe I'm wrong, though.
> 
> > Then I see (well no longer since the new year) use of "cool" noSQL thing
> > with no reason in a project that already had enough complexity.
> 
> I'm not sure what this refers to, to be honest. Image Warehouse used
> Mongo, if that's what you're referring to. That has been painful and
> we're ripping the whole thing out.

Sure Mongo.

> 
> > Then see big effort on using "cool and in" RESTful API in a situation
> > that really is about events.
> 
> Deltacloud and Conductor have been using REST (or at least something
> like it) for the past several years. Before I even joined the company a
> couple of years ago I was reading the Deltacloud documentation about its
> HATEOAS API.

The idea of a REST client needing no prior knowledge about how to
interact with any particular application or server beyond a generic
understanding of hypermedia may work for simple things.

Just an IMHO again. But look how painful is for Conductor to get around
places where the DC does not provide enough abstraction (e.g. error
states). No prior knowledge except of generic hypermedia? It's a dream
that fells apart then a more complex problem is to be solved. It's not
practical.

> 
> > But after the short time being on the team I am already tired of this
> > topic and as I said the architecture of Conductor seems to be a much
> > bigger problem for me then just the messaging between the components.
> > 
> > So that's all of my IMHO. Shoot me if you please.
> 
> Heh, in fairness, +1 to this -- I, too, am tired of this discussion.
> 
> -- Matt

Happy hacking!

-- 
Martin Povolny <mpovolny at redhat.com>
tel. +420777714458


More information about the aeolus-devel mailing list