Hello everyone. 

Periodically and seemingly at random our replicas crash with the above error. Dirsrv shows as stopped and restarting doesn't help.
Someone suggested earlier that this is due to problems with topology plugin but I don't think that the cause as we are still on
domainlevel=0.

I'm not sure if it's a problem with 389ds or with some other part of freeipa. The only other clue I can think of is that often we see inconsistencies
between replicas. IE a user that is supposed to be present everywhere goes missing on just one of the many replicas. 

I'm quite at a loss on how to troubleshoot this further. I hope that someone can assist.

ipactl start
Starting Directory Service
Failed to read data from service file: Failed to get list of services to probe status!
Configured hostname 'server.pop.domain.local' does not match any master server in LDAP:
No master found because of error: no such entry
Shutting down


cat errors
[26/Dec/2017:21:15:56.234793153 +0000] SSL alert: Sending pin request to SVRCore. You may need to run systemd-tty-ask-password-agent to provide the password.
[26/Dec/2017:21:15:56.236060353 +0000] SSL alert: Security Initialization: Enabling default cipher set.
[26/Dec/2017:21:15:56.236362922 +0000] SSL alert: Configured NSS Ciphers
[26/Dec/2017:21:15:56.236652729 +0000] SSL alert:       TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384: enabled
[26/Dec/2017:21:15:56.236921632 +0000] SSL alert:       TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA: enabled
[26/Dec/2017:21:15:56.237114079 +0000] SSL alert:       TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256: enabled
[26/Dec/2017:21:15:56.237317678 +0000] SSL alert:       TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA: enabled
[26/Dec/2017:21:15:56.237526365 +0000] SSL alert:       TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[26/Dec/2017:21:15:56.237746660 +0000] SSL alert:       TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA: enabled
[26/Dec/2017:21:15:56.237908539 +0000] SSL alert:       TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[26/Dec/2017:21:15:56.238087338 +0000] SSL alert:       TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA: enabled
[26/Dec/2017:21:15:56.238306056 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_256_GCM_SHA384: enabled
[26/Dec/2017:21:15:56.238517868 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_256_CBC_SHA: enabled
[26/Dec/2017:21:15:56.238724920 +0000] SSL alert:       TLS_DHE_DSS_WITH_AES_256_CBC_SHA: enabled
[26/Dec/2017:21:15:56.238889982 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_256_CBC_SHA256: enabled
[26/Dec/2017:21:15:56.239048124 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_128_GCM_SHA256: enabled
[26/Dec/2017:21:15:56.239233534 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_128_CBC_SHA: enabled
[26/Dec/2017:21:15:56.239402097 +0000] SSL alert:       TLS_DHE_DSS_WITH_AES_128_CBC_SHA: enabled
[26/Dec/2017:21:15:56.239767245 +0000] SSL alert:       TLS_DHE_RSA_WITH_AES_128_CBC_SHA256: enabled
[26/Dec/2017:21:15:56.239997083 +0000] SSL alert:       TLS_RSA_WITH_AES_256_GCM_SHA384: enabled
[26/Dec/2017:21:15:56.240177269 +0000] SSL alert:       TLS_RSA_WITH_AES_256_CBC_SHA: enabled
[26/Dec/2017:21:15:56.240376177 +0000] SSL alert:       TLS_RSA_WITH_AES_256_CBC_SHA256: enabled
[26/Dec/2017:21:15:56.240585031 +0000] SSL alert:       TLS_RSA_WITH_AES_128_GCM_SHA256: enabled
[26/Dec/2017:21:15:56.240745192 +0000] SSL alert:       TLS_RSA_WITH_AES_128_CBC_SHA: enabled
[26/Dec/2017:21:15:56.240897126 +0000] SSL alert:       TLS_RSA_WITH_AES_128_CBC_SHA256: enabled
[26/Dec/2017:21:15:56.241075071 +0000] SSL alert:       TLS_AES_128_GCM_SHA256: enabled
[26/Dec/2017:21:15:56.241245788 +0000] SSL alert:       TLS_CHACHA20_POLY1305_SHA256: enabled
[26/Dec/2017:21:15:56.241456256 +0000] SSL alert:       TLS_AES_256_GCM_SHA384: enabled
[26/Dec/2017:21:15:56.241617090 +0000] SSL alert:       TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256: enabled
[26/Dec/2017:21:15:56.241766851 +0000] SSL alert:       TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled
[26/Dec/2017:21:15:56.241947040 +0000] SSL alert:       TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256: enabled
[26/Dec/2017:21:15:56.249524586 +0000] SSL Initialization - Configured SSL version range: min: TLS1.0, max: TLS1.2
[26/Dec/2017:21:15:56.249909319 +0000] 389-Directory/1.3.5.10 B2017.102.203 starting up
[26/Dec/2017:21:15:56.261829771 +0000] default_mr_indexer_create: warning - plugin [caseIgnoreIA5Match] does not handle caseExactIA5Match
[26/Dec/2017:21:15:56.269563770 +0000] WARNING: changelog: entry cache size 2097152 B is less than db size 149151744 B; We recommend to increase the entry cache size nsslapd-cachememsize.
[26/Dec/2017:21:15:56.300878069 +0000] schema-compat-plugin - scheduled schema-compat-plugin tree scan in about 5 seconds after the server startup!
[26/Dec/2017:21:15:56.399266161 +0000] NSACLPlugin - The ACL target cn=automember rebuild membership,cn=tasks,cn=config does not exist
[26/Dec/2017:21:15:56.406444789 +0000] dna-plugin - dna_parse_config_entry: Unable to locate shared configuration entry (cn=posix-ids,cn=dna,cn=ipa,cn=etc,dc=domain,dc=local)
[26/Dec/2017:21:15:56.406758873 +0000] dna-plugin - dna_parse_config_entry: Invalid config entry [cn=posix ids,cn=distributed numeric assignment plugin,cn=plugins,cn=config] skipped
[26/Dec/2017:21:15:56.423696836 +0000] schema-compat-plugin - schema-compat-plugin tree scan will start in about 5 seconds!
[26/Dec/2017:21:15:56.434117007 +0000] slapd started.  Listening on All Interfaces port 389 for LDAP requests
[26/Dec/2017:21:15:56.434370916 +0000] Listening on All Interfaces port 636 for LDAPS requests
[26/Dec/2017:21:15:56.434602326 +0000] Listening on /var/run/slapd-domain-local.socket for LDAPI requests
[26/Dec/2017:21:15:56.517403933 +0000] slapd shutting down - signaling operation threads - op stack size 1 max work q size 1 max work q stack size 1
[26/Dec/2017:21:15:56.517944438 +0000] slapd shutting down - waiting for 28 threads to terminate
[26/Dec/2017:21:15:56.518216669 +0000] slapd shutting down - closing down local subsystems and plugins
[26/Dec/2017:21:16:01.429082375 +0000] Waiting for 4 database threads to stop
[26/Dec/2017:21:16:02.283796028 +0000] All database threads now stopped
[26/Dec/2017:21:16:02.302693986 +0000] slapd shutting down - freed 1 work q stack objects - freed 1 op stack objects
[26/Dec/2017:21:16:02.439672563 +0000] slapd stopped.