On 19 Mar 2020, at 04:08, thierry bordaz <tbordaz(a)redhat.com>
wrote:
On 3/18/20 1:51 AM, William Brown wrote:
>
>> On 18 Mar 2020, at 04:08, thierry bordaz <tbordaz(a)redhat.com> wrote:
>>
>> Hi William,
>>
>> I updated the design according to our offline exchange
> Thanks Thierry, I appreciate the conversation and the updates to the document: it
made clear there were extra details up in your brain but not in words yet :) it's
always hard to remember all the details as we write things, so thanks for the discussion.
Like you said, it's always good to have a team who is really invested and cares about
the work we do!
>
>
> Your design for the core server version looks much better! Thank you. I still think
there are some missing points. The reason to have a libpath rather than inbuild is to
avoid a potential linking to sssd/samba. I think also that the problem space of the global
catalog here needs to be looked at too. This feature is not in isolation, it's really
a part of that.
Okay, I will work on a new PR making core server able to retrieve/registers rewriters.
I think the "need" to improve the usability of rewriters is not specific to
global catalog. Global Catalog is just an opportunity to implement it. I think parts of
slapi-nis, integration of vsphere, GC (and likely others) are also use case for rewriters.
They were implemented in different ways because rewriters were not easy to use or simply
not known.
Yes, that's perfectly reasonable, and shouldn't stop your idea from being created
- what's concerning me is that without a full picture you don't know how far to
take these rewriters or what direction, or what might be needed.
>
> This means we have a whole set of deployment cases to look at.
>
> So the deployment will look like:
>
> IPA DS --> IPA GC
>
>
> So an ipaAccount from the IPA DS instance will be "copied and transformed"
into the IPA GC. This process is as yet undefined (it sounds like it may be offline or
something else ...). We are simply not dealing with one instance now, but an out-of-band
replication and transformation process. It's unclear whether the data transform is
during this loading process, or in the IPA GC somehow.
>
> From what I understand, it sounds like a method to take an ipaAccount and transform
it to an AD GC account stub. Then inside of that IPA GC there are some virtual attributes
you wish to add like objectSid binary vs string representations, objectCategory, maybe
others.
>
> So from our discussion, we have currently focused on "how do we transform
entries within a single directory server". But that's not the problem here. We
are saying:
>
> "We take an entry from IPA DS, transform it to an IPA GC stub entry, and then
apply a set of further "in memory" transformations"
One of the biggest issue with GC is schema. IPA DS and IPA GC have not compatible schema.
They can not be in the same replication topology.
So provisioning of IPA GC requires transformations rules to present an other
"view" of IPA DS data. Those transformations will be on the write path (i.e.
stored in DB/indexed). This transformation work is almost done and is completely
independent of 389-ds.
All of this is "write" path: provisioning (online or offline) and
transformation.
The problem for IPA GC is now on the "read" path. AD clients are use to smart
shortcuts/control that are supported by IPA GC.
This is the IPA GC instance that will register the rewriters to act as GC does.
Yep, I'm aware :)
>
> If that's the process, why not do all the transforms as required in the DS ->
GC load process? You raised a critically key point - we have a concern about the write
path as the transform point due to IO or time to do the transform, but it sounds like you
have to do this anyway as an element of the DS -> GC process.
Some of the transformation rules, on the write path, are quite complex. Looking at
slapi-nis config entries gives an idea what is needed. In addition to those
transformations, DS to GC online provisioning is not simple at all. Relying on sync-repl,
you then need to transform a received entry into an update. At the moment it is an offline
provisioning via transformation and import (much simpler).
To be honest I am afraid that the transform rules will result in rewriting slapi-nis.
*puts finger on nose* I do not want to be near that toxic rewrite at all.
>
> I think everytime I have spoken to you about this, I have kept learning more and more
about this, and the more I see, I have many concerns about this feature. I think we do not
have the full picture. You have admitted that you don't know the full extend or ideas
here. There is clearly a communication break down here to our team from the IPA project,
and they aren't telling us what they want. It sounds like they are asking you to just
do "a small piece" but only they know the bigger picture.
>
> The IPA project has the following designs:
>
>
https://www.freeipa.org/page/V4/Global_Catalog_Support
>
>
https://www.freeipa.org/page/V4/Global_Catalog_HLD
>
>
https://www.freeipa.org/page/V4/Global_Catalog_Access_Control
>
>
https://www.freeipa.org/page/V4/Global_Catalog_Data_Transformation
>
> This also links to:
>
>
https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows...
>
> The freeipa design pages are extremely shallow on details. The entire section on how
they plan to get data into the GC is:
>
> """
> Global Catalog provisioning
>
> The data in Global Catalog is provisioned from the primary LDAP server instance
running on the same FreeIPA master. A SYNCREPL mechanism is used to retrieve the changes
and a modified slapi-nis module is used to transform FreeIPA original data to a schema
compatible with Global Catalog in Active Directory. Unlike the original slapi-nis module,
the data is stored in a proper LDAP backend so it is persistent across the directory
server restarts.
> """
You are right I do not know the big picture. What I know is that parts of GC needs can be
solved with rewriters that is by the way a supported 389-ds interface. So storing
rewriters in simple shared library rather than in plugins will help both IPA and 389-ds.
Without the big picture we don't know what they will ask from the rewriters, and what
we can or cannot deliver.
>
> Where is the example config? Proof of concept? Even a conceptual set of accounts and
groups showing the data transformation? How will they synthesise stable object data
points?
>
> The section of "data transformation" even goes to a blank page. Is the
rewrite you are being asked to do just for objectSid once all these other transforms are
done? Or is there more?
>
>
> Honestly, it's worth reading the "how global catalog works" from msdn.
Just to put it in contrast, that document (when converted to a pdf) is 61 pages long. Look
at the features. Group caching, GC replication, partialAttribute replication based on
schema, more ...
>
>
> Honestly, Thierry, I trust you as a very smart and capable engineer, but you do not
have the full picture here - none of us do. This seems like a feature that will explode in
complexity and scale, and if not done *properly* from the start, may end up with many many
half-baked, poorly designed solutions tacked together to make it look like it works. And
that means we'll end up carrying that burden, just like slapi-nis (which is everyones
favourite plugin ...)
Again, rewriters is not new. It has been a supported interface for years. The design is
just to make them simpler to develop/deploy.
Looking at some plugins I think they are related to a way to give different
"views" of the same dataset. Many time, a rewriter, specific to ldap client
needs is a good option.
If GC can make use of it great. But I am sure that others (like vsphere) will appreciate.
That's not the problem. You are right that having improved rewriter support, probably
has some good options for other plugins, or other areas. The issue is without the bigger
picture, we don't know what they need. We don't know what we are on the hook for.
Let's be clear, to me as an external person, a core team of the 389 project, the
information in those design documents is not enough for me to make informed engineering
decisions about this feature.
There is a pattern and history to this behaviour.
I think what's really concerning isn't the technical issues, but the social. I
want to make clear - You, Thierry, yourself admitted you do not know what is fully
expected of you in this feature. How are you, as an engineer meant to do your best
possible work, without the full picture. You are very smart, but not psychic last time I
checked :)
This does not make me comfortable.
I also know - having been inside Red Hat, and now external to it, that the FreeIPA team
does a lot of discussion internally and privately. It needs to stop. If they want to
request features of our project, they need to accept that 389-ds has upstream, core team
members who are not part of Red Hat. They need to be engaging on 389-devel, not coming to
you internally, and asking for features. They need to be designing their features,
publicly, and clearly, in detail, so that we can make informed engineering decisions for
our project, that includes our full team and community.
I have resisted talking about this publicly for a long time, but I think it's time
that it's taken to the open - the FreeIPA team has communication and social challenges
that they need to address. While these social issues continue, we will continue to see
poor quality features being churned out that negatively impact our users, and our
reputation as a project.
At this point, I believe that this rewriter feature can not progress until the FreeIPA
project puts forward a complete, detailed and well constructed design of "what they
require" for their global catalog feature.
Thanks
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs