On 06/06/2014 05:34 PM, Dmitri Pal wrote:
On 06/06/2014 06:46 AM, Petr Viktorin wrote:
> On 06/06/2014 04:09 AM, Dmitri Pal wrote:
>> Over the time multiple tools have been developed and used to address
>> different use cases around SSSD configuration.
>> We now have:
>> 1) SSSD that needs to read a configuration. It uses ding-libs
>> libini_config for that purpose.
>> libini_config was historically developed to read configuration from C
>> services like SSSD.
>> It is currently used by SSSD and GSS proxy
>> It is not capable of modifying and saving the configuration. To be fair
>> it is capable but the interfaces are not exposed at the high level API.
>> It will take some (couple days or so to close this gap).
>> 2) SSSD also needed to create sssd.conf at the installation time. It is
>> ueasy to do it from Python this is why an SSSD config python script have
>> been created. It is good at modifying the configuration and setting
>> specific values at the beginning of the sssd.conf's life. It is not
>> suited well (at least yet) for long term maintenance of the
>> configuration file. It is also SSSD specific.
>> 3) Over the time there were RFEs that suggested that sssd configuration
>> should be extensible or overwritable.
>> The following ticket cover that:
>> The design has been published here
>> 4) There is a desire to manage SSSD over OpenLMI a cockpit in an object
>> oriented way. To do that (following the standard LMI model) the
>> following layring has been suggested and implemented:
>> Consumer (OpenLMI/Cockpit) -> D-Bus interface -> ifp service (responder)
>> -> augeas c api
>> More details are here:
>> 5) In the absence of the good configuration solution for the integration
>> purposes, i.e. to integrate applications running on top of the system
>> with SSSD as a short term solution a puppet-augeas based way of
>> configuring sssd.conf was introduced the other day. The main
>> requirements are to be able to modify existing configuration and more
>> specifically change the values that represent a list of values.
>> 6) From the very beginning we were hoping that there would be a way to
>> be able to define the grammar of the configuration file and check it to
>> be able to diagnose the config files and prevent their corruption. The
>> idea was to add this capability to libini as it provides a set of nice
>> convenience capabilities in comparison to Augeas. Also Augeas until not
>> long ago was not included in some distors.
> Could you elaborate on the convenience capabilities?
Last time I looked at Augeas (which was some time ago) it did not:
- allow multiple key instances like this
key = val1
key = val2
and ability to iterate a section and access them.
This feature is used by GSS proxy
- allow processing the config and handling the sections in different
ways detecting collisions, erroring on them or merging them on the fly
as one requests
- merging two configs the way you want
- support of BOMs - encodings at the beginning of the file
- adding and or modifying comments
- allowing to wrap values
- processing lists of values in a convenient way
Some of these are explicit non-goals of Augeas; some are features they
might have or want to have.
In other words, yes, we will need to do some work if we adopt Augeas.
Some of that work would need to be be on top of Augeas.
May be some of these feature already available and I am just not
I guess some research is in order.
Also with libini there is a lot of control that we have to add
on the pace we need.
> Is it more work to complete libini, or add convenience capabilities
> to/on top of Augeas?
libini has continence capabilities. It will be more work and AFAIU
feature degradation if we replace one with another but may be augeas is
more mature than I remember it to be.
> Another factor is how many projects will benefit from SSSD devs
> working on libini vs. on Augeas.
This is hard to say. We already did this evaluation once and said Augest
is not good enough for us so we will go with libini.
But I started this discussion to make a conscious choice moving forward.
Obviously the choice is yours, I'm not even a SSSD developer. I'm just
providing some arguments to weigh.
>> The list above presents a spectrum of use cases and solution
>> use cases.
>> The goal of this email is to try to enumerate the use cases and
>> constraints and understand how the solution for those use cases can be
>> consolidated and unified and where it makes sense todo and where not. It
>> is a goal to define a reasonable alignment between the technologies and
>> provide a path forward to a reusable and maintanable stack.
>> So we have the needs:
>> 1) Read config files at run time from C (daemons and D-BUS provider)
>> 2) Write config files from C (configuration interface over LMI)
>> 3) Create and modify files at installation and upgade using some simple
>> mean/API which can be C, python or anything else
>> 4) Be able to manage configuration files in a object oriented manner
>> with or without D-BUS running
>> 5) Be able to augment the code file without touching it
> I don't understand this sentence, could you clarify?
>> 6) Provide a way to validate the file
>> So I see the following layering
>> | High level tools |
>> | Object oriented interface |
>> | |
>> \/ \/
>> +-----------------+ +------------------+
>> | D-BUS responder | | D-BUS shim layer |
>> +-----------------+ +------------------+
>> | --------------------|
>> | |
>> \/ \/
>> +-------+ +----------------+
>> | C API |<-------------| File validator |
>> +-------+ +----------------+
>> /\ \
>> | \ +--------------+
>> | -------------->| Config file |
>> | +--------------+
>> | /\
>> +----------------+ |
>> | SSSD/GSS proxy | |
>> +----------------+ |
>> | SSSD Coinfig API |
>> | Simple tools |
>> This layout shows three distinct paths towards config file:
>> 1) Old python API and tools on top (does authconfig use this path?)
>> 2) Top level D-BUS api and advanced object oriented tools
>> 3) C API
>> We need to make several decisions at this point:
>> a) How much we enhance-update-improve existing python API instead of
>> using new D-BUS interface.
>> IMO we should leave python API alone for foreseeable future but any new
>> changes should go via the new D-BUS interface
>> b) How we make sure that the D-BUS interface is really usable. It is not
>> that light weight.
>> Would be nice to be able to have a shim layer for the cases like docker
>> where there might not be D-BUS or systemd. This however might not be a
>> problem for us to solve. It might be a generic problem for all LMI style
>> providers. A lot in D-BUS world is autogenerated so I suggest that we
>> ask to be able to generate a mini D-BUS interface that would go directly
>> from the interface to the logic implemented in responder. I do not know
>> what is best. But I think it is a discussion to have with D-BUS guys and
>> should be more general.
>> For now we can assume that D-BUS is there and docker use case will be
>> solved in some generic way.
>> c) If we assume D-BUS and systemd it really makes sense to separate the
>> responder in charge of configuration from other responders and make it
>> started on demand when there is a real call over the D-BUS to make a
>> config change. IMO this work for splitting ifp into data ifp and config
>> ifp should be filed as a ticket and done as a part of 1.13 release.
>> d) What is going to be the low level C library? We have two candidates
>> libini and augeas.
>> I definitely have a preference towards libini but this does not mean I
>> am right. I do not have a lot of time to maintain libini. I do not see a
>> reason to rip it out and replace with augeas yet. But I also do not see
>> a reason why we should rely on augeas. Right now IMO it would be more
>> work to rip libini and switch to augeas but over time if we do more and
>> more augeas work it might be much less. I feel people are more
>> comfortable with augeas than using libini and adding to it as they go. I
>> do not know why but suspect it is because augeas has appeal of a more
>> commonly used tool/interface. It is all about what we think is the right
>> path. If we decide that augeas is the way to go then we should not
>> invest into enhancing libini and start building our improvements in
>> augeas or around augeas. If we think that libini is the right path
>> because it fill give us more features, flexibility and control then let
>> us make it so that I am not the only contributor to it and people would
>> not have problems throwing patches at it on as needed basis.
>> For me this is a very important decision. Right now I feel I am on hook
>> to continue investing my scares time into libini. I think I am holding
>> people back with this so I would rather come to terms and move forward
>> whatever we think is best for SSSD, GSS proxy and potential other
>> daemons that need to read and deal with config files.
> It seems the SSSD devs here in Red Hat's Brno office don't have the
> courage to say this, so let me convey the general idea floating around
> here: continuing with libini is a bad idea.
> We're in the business of collaboration, not writing our own little
> libraries where the main advantage is having control over the project.
I would agree with you if we were at the decision point several years ago.
libini now is mature and can do what we need. It is stable.
Adding to it is simple and not much effort. I can prove it and turn back
pretty much all the code we need for it (except grammar validation) to
address writes, merging etc. But I can't support it. It is not
sustainable in a long run.
That's the point.
Augeas is sustainable in the long run, because it's, AFAICS, a healthy
And anything useful built on top of Augeas has a better chance to get a
community. (Or be replaced with a more popular Augeas-based
implementation of the same feature, of course.) That's the way things go.
The grammar validation IMO should be a generic feature and can be
potentially another layer that processes config and check if it is OK.
It can be independent of augeas/libini but making it generic would be
more work than really needed so it should rather pick one and target it.
If you want a generic feature, you'll need to base it on a tool that
other people use, not on a tool that only caters to the needs of two
projects. Otherwise the feature will never be generic, because it will
also only cater to those two projects. And if the two projects happen to
not have many outside contributions, neither will the feature.
Of course, having a non-generic feature is possible, but you'll have to
write it and support the little tower of code forever. From the ground
up it'd be incompatible with what other people use, and there'd be no
integration at the higher levels because of the "making things generic
is more work" philosophy.
Now here is the situation - we have to pick one. I look at how much
there is for me to address everything that needs to be addressed in
libini to meet the requirements of the stack above and I say days
(though I do not have those days). I look at augeas and I say weeks or
months. But this is my take because I know libini and invested into it.
Well, you don't have the time, so ask the person who'll need to do the
changes for an opinion.
Or actually don't ask the developer; developers' estimates always fall
short. Especially when they know the code :)
We already have libini. We already support it and it has a lot of
features. I am fine replacing it if we are sure that augeas would not
prevent us from doing what we are doing now. If that would require a
major refactoring and it would require us to contribute to augeas
significant chunks of code I do not think it is justified.
> Contributing to Augeas will help many more people than playing in our
> own sandbox.
True in general case. But we are not altruists here. Everything has cost
So far I have not seen any real technical argument why augeas is a
better choice. I see a generic statement about general value to open
source that is not measurable. Let us be practical. Give me the value
and the cost and convince me that this is the right cost to have.
Look, you're the manager here. Cost and benefit analysis is your turf; I
don't have the capability of measuring ideas by money.
I am, however, an employee of a company that prides itself on
collaboration. I am used to working with others to solve common
problems. Augeas clearly solves many of the same problems libini does,
it's fairly popular and has some people who clearly know what they are
We are not altruists. We are collaborators, because working with others
is better for us in the long run.
I can be convinced otherwise I would not have had this conversation
not with the argument above.
> And it'll also be better for us -- I don't think libini is getting
> Ruby bindings or a big tool ecosystem any time soon.
Do we need ruby bindings? Is it really a use case we care?
Well, that's for the manager to decide. I can see two possible answers:
* No, it is not.
We don't care about anything that doesn't benefit us directly.
This is exactly why there will never be a community around libini.
* Yes, it is.
There's a module for Puppet that uses these Ruby bindings to configure
AFAIK, neither Puppet nor Augeas authors needed to write the
integration. It came from the community.
By joining the Augeas community, we'd get Puppet integration for SSSD
configs, right now.
And later, we'd automatically get similar integration for whatever tools
are popular enough for someone to write a generic Augeas plugin for.
> If there is anything missing in Augeas then we should extend
> instead of trying to make another implementation.
We already have implementation this is the point and the question is: is
it worth dropping it or not.
Convince me that it is worth.
On 06/06/2014 05:47 PM, Dmitri Pal wrote:
So let me summarize pros and cons:
Pros of libini
- already working and stable parser for INI files
- already integrated with SSSD and GSS proxy
- has advanced convenience features?
- can be turn around and delivered very fast because of control
Cons of libini
- zero community
- INI files only
- main developer does not have time to work on it
Pros of augeas
- has parser
- has community
- contributing to it will have general benefit
Cons of augeas
- not integrated into SSSD and GSS proxy
I'm sure you heard more than once that we need to integrate things,
rather than build separate stacks.
- does not have advanced features (to be verified)
- less control and thus slower turn around
Yes, unfortunately, collaboration means having to do politics. It's the
way it works in open source, except maybe in our little island of
open-source projects with no community.
Anyway, control is primarily based on the time you invest into the project.
If push comes to shove, you can always fork it, or carry your own stack
of patches for absolute control.
So when I look at it for me it makes perfect sense to continue
investing into libini to address its cons, i.e. if we contribute to it we will foster a
community for it.
Umm, how good are we at fostering developer communities?
Can you create a sustainable community around libini? Is it actually
useful for use cases we don't care about?
If you want a *community* around the config parsing library we'll use, I
can tell you without doubt that porting to Augeas will be WAY easier
than attracting a community to libini.
A healthy community would also mean that *we* would not really be in
control. One of libini's big advantages would be lost.
Then we already leverage the functionality we have, work for greater
good, do not waste cycles and show leadership. It also allows us to be focused on the use
cases we really care about.