CLI on server part 2

Greg Hinkle greghinkle at gmail.com
Wed Jan 12 14:29:24 UTC 2011


On Jan 12, 2011, at 8:25 AM, Lukas Krejci wrote:

> Thanks Greg for the explanations, they were very helpful.
> 
> On Tuesday, January 11, 2011 18:44:38 Greg Hinkle wrote:
>> On Jan 11, 2011, at 12:20 PM, Alan Santos wrote:
>>> Thanks Greg, I appreciate the history, it's very helpful.
>>> 
>>> You're absolutely correct.  High speed event storage needs to be
>>> considered separate from content storage: two very different use-cases
>>> and requirements.      Content management without something like
>>> modeshape to offload metadata management, to abstract the storage, to
>>> simplify versioning and expand the connectivity choices is a nightmare. 
>>> With a content/metadata repository - like modeshape - you don't have to
>>> worry about the RDBMS being out of sync with the SCM because that is its
>>> purpose.
>> 
>> It is very likely that you will still need each and every version, along
>> with all metadata stored in the relational store where you can query it
>> and join it. For example, tracking the request and actual installation of
>> a bundle (of a specfic version) to a specific resource. That bundle
>> version will end up needing a representation in the relational database
>> and you'll end up with the content store doing little more than holding
>> the bits. If you have no physical row in the db then you've just created a
>> relational model without referential integrity. Where does access control
>> get stored? How can you tell which resources a particular bundle version
>> is installed to. Versioning need be relational in this case leaving the
>> only benefit of a content store being the modes of access, which are
>> actually easier to do other ways than moving the content itself.
>> 
> 
> I'd argue that content store's purpose is to hold the bits and I actually 
> wouldn't want it to do much more than that (maybe store metadata extracted 
> from the bits). 
> 
> Your reasoning about individual versions of bundles needing to be relational 
> is very true. But still it doesn't persuade me that the actual bits belong in 
> the DB.
> 
> Bundles, scripts, and after all the content packages (as refered to in the 
> content subsystem) are in the end "owned" by our users and not by RHQ. It is 
> their data and is subject to their own workflows (well, if not their own then 
> at least someone else's (think CSP)). By storing the actual bits in our 
> database, we don't add anything new to their workflows, we just create another 
> "cache" of the data. I think this is completely unnecessary and, especially in 
> case of bundles, puts unneeded (if bareable) load and requirements on our 
> database. I mean once the provisioning subsystem is able to provision whole 
> operating systems, do we really want to store the whole OS image (in each and 
> every version it happens to be used) in our database?

That's why there was a separation between "Content Sources" and our internal 
package management. The content sources could pull packages for deployment 
from YUM, SCMs, WebDAV or whatever, but once its in our system we wanted to 
be able to track it fully and reliably. (very important for compliance)

The reason this internal store is necessary is because its critical to have exactly 
the bits that get pushed out. That means tracking the actual bits. If someone 
comes and updates the bits in a version in the external repository that doesn't 
mean those are the bits that are now deployed onto machines associated with 
that version. So, the way I see it, RHQ does own the bits of what is actually 
deployed. It should integrate with existing stores for its sources, but once the 
packages go out... or are discovered, RHQ is the authority on which version 
is where and what that version really is.

I'm not sure how basic bundles would have much of a performance impact. We 
were storing all the packages for many machines (the entire OS) without much 
trouble in the content store. Gigabytes of storage and thousands of packages 
per machine where never really a concern with that architecture.


> 
> I must say that I don't completely understand what you said about modes of 
> access in the last sentence. If you meant that providing access to the data 
> stored by RHQ is easier by other means than moving the data away from RHQ then 
> you are certainly correct, but I think it's a wrong point of view. I think 
> that we have to look at the data as naturally being "foreign" to RHQ and us 
> only referencing it. RHQ should not be considered the source of the data, 
> maybe a proxy (but that doesn't mandate us storing the actual bits).

This was just regarding integrating RHQ into an environment by letting the user 
use clients they were familiar with or via integration with existing content stores.

If a customer puts all their release builds into an SCM we could pull from there 
to allow deployment. If they don't have that they could use a WebDAV client or 
and SCM client to push the bits to us, though I'm not sure how necessary that is.

My point was that this was practically the only advantage a typical CMS would 
end up giving you since you were going to be required to rebuild the version 
tracking and metadata and everything else in the relational store. (As has 
already  been done in the case of both Packages and Bundles)

> 
> 
>>> Granted, you get some of those requirements for free by using an RDBMS,
>>> but you still end up fighting significant battles -  e.g. versioning and
>>> non-sql based access (I've seen your webav work), only now you are doing
>>> so with increased complexity, impedence mismatch and loss of
>>> transparency.
>>> 
>>> It's a hard problem to solve no matter how it's approached, using the
>>> wrong tool makes it harder to do so.
>>> 
>>> -alan
>>> 
>>> On 1/11/11 11:31 AM, Greg Hinkle wrote:
>>>> In other words this data is linked to other parts of the persistence
>>>> model. Referential integrity is important as is transactional access.
>>>> If you store a bundle, script or package in an on-disk SCM how do you
>>>> ensure referential integrity to the installed version, the resource its
>>>> deployed to, etc.? Then consider backups, clustering and recovery...
>>>> right now it's as simple as having a valid RDBMS backup strategy, but
>>>> as soon as you introduce a separate store you'll have to back that up.
>>>> Then what happens if your SCM get's out of sync with the related
>>>> records in the DB? If you cluster with a cluster file system then you
>>>> have to install, monitor and maintain that store and deal with network
>>>> setup, host OS differences, etc.
>> 
>> _______________________________________________
>> rhq-devel mailing list
>> rhq-devel at lists.fedorahosted.org
>> https://fedorahosted.org/mailman/listinfo/rhq-devel
> _______________________________________________
> rhq-devel mailing list
> rhq-devel at lists.fedorahosted.org
> https://fedorahosted.org/mailman/listinfo/rhq-devel



More information about the rhq-devel mailing list