On 19 Nov 2020, at 20:34, Graham Leggett <minfrin(a)sharp.fm>
wrote:
On 13 Nov 2019, at 12:29, Graham Leggett <minfrin(a)sharp.fm> wrote:
> On 13 Nov 2019, at 01:37, William Brown <wbrown(a)suse.de> wrote:
>
>>> Does anyone know why 389ds would suddenly stop sending the full certificate
chain while replicating?
>>>
>>> It also looks like the error handling in 389ds SSL is broken - if the slave
sent “unknown CA" to the master, the master needs to log that fact, and not report
the error as “success”.
>>
>> We'll need to see the output of certutil -L -d
/etc/dirsrv/slapd-<instance>/ from both the master and replica servers please.
>>
>> In a TLS auth process the client doesn't send it's CA - if you get
unknown CA it's most likely the replica has either had the CA and it's chain
members expire, or they are not marked as trusted for client auth. So that's why
I'd like to see the certutil output please.
>
> I discovered the same problem had been reported in OpenLDAP:
https://www.centos.org/forums/viewtopic.php?t=67042
>
> This in turn is caused by a regression in NSS, where it is no longer sufficient to
have a trusted root certificate, you now need all intermediate certificates marked as
trusted as well.
>
> Making the following change to the intermediate certs fixed the problem:
>
> [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>
> Certificate Nickname Trust Attributes
> SSL,S/MIME,JAR/XPI
>
> intermediateB ,,
> intermediateA ,,
> rootrootroot CT,C,C
> ldap01 u,u,u
> [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
"intermediateA"
> [root@ldap01 ~]# certutil -M -d /etc/dirsrv/slapd-hg -t "CT,C,C" -n
"intermediateB"
> [root@ldap01 ~]# certutil -L -d /etc/dirsrv/slapd-hg
>
> Certificate Nickname Trust Attributes
> SSL,S/MIME,JAR/XPI
>
> intermediateA CT,C,C
> intermediateB CT,C,C
> rootrootroot CT,C,C
> ldap01 u,u,u
>
> Raised the bug here:
https://bugzilla.redhat.com/show_bug.cgi?id=1771979
Coming back to this one - got to the bottom of this while investigating something else
that wasn’t working.
This wasn’t a regression in NSS, but rather a regression in the openldap libraries
shipped by RHEL7.5 and above.
For reasons that I haven’t found, there was an architecture change made half way through
the RHEL7 lifecycle where openldap was linked to openssl instead of NSS.
Openldap's NSS support and openldap’s openssl support differ in a fundamental way -
with NSS, when openldap makes an SSL connection intermediate certificates are filled in by
the client side as normal. With openssl, when openldap makes an SSL connection
intermediate certificates are ignored, and the connection breaks.
The hack workaround above fixes this because openldap’s openssl support expects you to
place intermediate certs in your trusted certificate store. As soon as you mark the
intermediates as trusted in NSS, the hack workaround in 389ds that makes replication
sort-of work bound to two different crypto libraries exports trusted certs across into the
ca certificate list passed to openldap. Openldap then finds the intermediates and things
work.
Fundamentally there are two bugs:
-
https://bugzilla.redhat.com/show_bug.cgi?id=1898924
- An architectural change half way through the lifecycle of what is supposed to be a
stable OS.
I seem to remember this change (this was at a time I worked at RH). If memory serves
correctly, OpenLDAP upstream removed/deprecated their NSS support. This was making it much
much harder to apply fixes for issues both stability and technical, so moving to OpenSSL
was the "best move" for customer support.
Even internally to 389-ds which has to link to OpenLDAP for some outbound client
operations, it internally swapped from NSS to OpenSSL for this as well, which involves
extracting some certificates into temporary stores for OpenLDAP client to use. It's
quite fun to put it mildly.
There are very good reasons why those decisions were made, and that was very carefully
managed too. It was a lot of work and ultimately, it did make the OpenLDAP client library
better for our maintainers and many consumers, but as you have noticed, these are complex
systems, designed and built in a time that preceded deep testing of things. I believe our
client TLS auth tests were made *after* the OpenLDAP to OpenSSL switch was made, which
could be why this was not noticed.
Anyway, I hope that this gives some more context to why this happened.
—
Sincerely,
William Brown
Senior Software Engineer, 389 Directory Server
SUSE Labs, Australia