The server uses the openldap client libraries for replication connections.  Setting nsslapd-ssl-check-hostname sets these flags on the connection as follows:

For server authentication it sets this flag:

    LDAPSSL_AUTH_CNCHECK   --> This checks the hostname in the certificate subject to that of the host
 
For SSL client auth it sets this flag:

    LDAP_OPT_X_TLS_HARD

Okay so it does more than is documented, now I get it.

So the issue here is either openldap is not finding the correct hostname, or the hostname in the certificate subject is wrong.

As I stated previously, my domain name and cert is good. Even the reverse dns record is correct.

I tried replacing the certificate with an incorrect one (with invalid CN) and the error displayed in log is the very same. So yes, it looks like “something” does not match (but what?)

Connecting to ldap server itself works, even openssl s_client verifies the server cert ok (including the chain, what was a nice surprise to me).

Just to be clear: I’m using my own root CA, with an intermediate CA which issued cert for CN=ldap-master-b01.example.com and CN=ldap-master-b02.example.com. Both are imported into certstore with nickname “CN=ldap-master-b0[12]” (including the “CN=“). 

In cn=RSA,cn=encryption,cn=config, I use nsSSLPersonalitySSL='CN=ldap-master-b[01].example.com’.

I tried changing the errorlog-level as you suggested, but I got no better message than...

[09/May/2018:21:13:25 +0200] NSMMReplicationPlugin - agmt="cn=rw-to-ldap-master-b02.example.com" (ldap-master-b02:636): binddn = cn=MasterMasterReplicationManager,cn=config,  passwd = {AES-TUhNR0NTcUdTSWIzRFFFRkRUQm1NRVVHQ1NxR1NJYjNEUUVGRERBNEJDUXhZamN5WXpNeVppMDNPR00zTXpOaA0KTUMxaE1XTmtabUl5WmkwMVpUVmtOR1l5TlFBQ0FRSUNBU0F3Q2dZSUtvWklodmNOQWdjd0hRWUpZSVpJQVdVRA0KQkFFcUJCRDBuaTFJaDRXMmZDcnlqWUtXQmlMRw==}yFb3FVwDwpWKupgUWiS4wg==
[09/May/2018:21:13:26 +0200] slapi_ldap_bind - Error: could not send bind request for id [cn=MasterMasterReplicationManager,cn=config] authentication mechanism [SIMPLE]: error -1 (Can't contact LDAP server), system error -5987 (Invalid function argument.), network error 115 (Operation now in progress, host "ldap-master-b02.example.com:636”)

root@ldap-master-b01:~# host ldap-master-b02.example.com
ldap-master-b02.example.com has address 100.127.177.145
root@ldap-master-b01:~# host 100.127.177.145
145.177.127.100.in-addr.arpa domain name pointer ldap-master-b02.example.com.

root@ldap-master-b02:~# certutil -L -d /etc/dirsrv/nss/ -n "CN=ldap-master-b02.example.com"|grep Subje
        Subject: "CN=ldap-master-b02.example.com"