Ok, here I go again -  this does not make sense. Looking at this topology - but for a moment, ignore IPAP1, as that is the one I an trying to add:


The problem is - IPAC1 is on the other side of a firewall from IPAP1, and only IPAC is permitted to talk to it, but that should not be a problem.

When I add IPAP1 in as a replica, it gets as far as:

Continue? [no]: yes
Run connection check to master
Connection check OK
Configuring NTP daemon (ntpd)
  [1/4]: stopping ntpd
  [2/4]: writing configuration
  [3/4]: configuring ntpd to start on boot
  [4/4]: starting ntpd
Done configuring NTP daemon (ntpd).
Configuring directory server (dirsrv). Estimated time: 30 seconds
  [1/40]: creating directory server instance
  [2/40]: enabling ldapi
  [3/40]: configure autobind for root
  [4/40]: stopping directory server
  [5/40]: updating configuration in dse.ldif
  [6/40]: starting directory server
  [7/40]: adding default schema
  [8/40]: enabling memberof plugin
  [9/40]: enabling winsync plugin
  [10/40]: configuring replication version plugin
  [11/40]: enabling IPA enrollment plugin
  [12/40]: configuring uniqueness plugin
  [13/40]: configuring uuid plugin
  [14/40]: configuring modrdn plugin
  [15/40]: configuring DNS plugin
  [16/40]: enabling entryUSN plugin
  [17/40]: configuring lockout plugin
  [18/40]: configuring topology plugin
  [19/40]: creating indices
  [20/40]: enabling referential integrity plugin
  [21/40]: configuring certmap.conf
  [22/40]: configure new location for managed entries
  [23/40]: configure dirsrv ccache
  [24/40]: enabling SASL mapping fallback
  [25/40]: restarting directory server
  [26/40]: creating DS keytab
  [27/40]: setting up initial replication
Starting replication, please wait until this has completed.
Update in progress, 3 seconds elapsed
Update succeeded

  [28/40]: adding sasl mappings to the directory
  [29/40]: updating schema
  [30/40]: setting Auto Member configuration
  [31/40]: enabling S4U2Proxy delegation
  [32/40]: initializing group membership
  [33/40]: adding master entry
  [34/40]: initializing domain level
  [35/40]: configuring Posix uid/gid generation
  [36/40]: adding replication acis
  [37/40]: activating sidgen plugin
  [38/40]: activating extdom plugin
  [39/40]: tuning directory server
  [40/40]: configuring directory to start on boot
Done configuring directory server (dirsrv).
Configuring Kerberos KDC (krb5kdc)
  [1/5]: configuring KDC
  [2/5]: adding the password extension to the directory
  [3/5]: creating anonymous principal
  [4/5]: starting the KDC
  [5/5]: configuring KDC to start on boot
Done configuring Kerberos KDC (krb5kdc).
Configuring kadmin
  [1/2]: starting kadmin
  [2/2]: configuring kadmin to start on boot
Done configuring kadmin.
Configuring directory server (dirsrv)
  [1/3]: configuring TLS for DS instance
  [2/3]: importing CA certificates from LDAP
  [3/3]: restarting directory server
Done configuring directory server (dirsrv).
Configuring the web interface (httpd)
  [1/22]: stopping httpd
  [2/22]: setting mod_nss port to 443
  [3/22]: setting mod_nss cipher suite
  [4/22]: setting mod_nss protocol list to TLSv1.0 - TLSv1.2
  [5/22]: setting mod_nss password file
  [6/22]: enabling mod_nss renegotiate
  [7/22]: disabling mod_nss OCSP
  [8/22]: adding URL rewriting rules
  [9/22]: configuring httpd
  [10/22]: setting up httpd keytab
  [11/22]: configuring Gssproxy
  [12/22]: setting up ssl
  [13/22]: configure certmonger for renewals
  [14/22]: importing CA certificates from LDAP
  [15/22]: publish CA cert
  [16/22]: clean up any existing httpd ccaches
  [17/22]: configuring SELinux for httpd
  [18/22]: create KDC proxy config
  [19/22]: enable KDC proxy
  [20/22]: starting httpd
  [21/22]: configuring httpd to start on boot
  [22/22]: enabling oddjobd
Done configuring the web interface (httpd).
Configuring ipa-otpd
  [1/2]: starting ipa-otpd
  [2/2]: configuring ipa-otpd to start on boot
Done configuring ipa-otpd.
Configuring ipa-custodia
  [1/4]: Generating ipa-custodia config file
  [2/4]: Generating ipa-custodia keys
  [3/4]: starting ipa-custodia
  [4/4]: configuring ipa-custodia to start on boot
Done configuring ipa-custodia.
Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    Timed out trying to obtain keys.
ipa.ipapython.install.cli.install_tool(CompatServerReplicaInstall): ERROR    The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information

and the ipareplica-install.log shows:

2018-02-28T11:52:23Z INFO Waiting up to 300 seconds to see our keys appear on host: ipac1
2018-02-28T11:54:30Z DEBUG Transient error getting keys: '{'desc': "Can't contact LDAP server"}'

2018-02-28T11:58:47Z DEBUG The ipa-replica-install command failed, exception: RuntimeError: Timed out trying to obtain keys.
2018-02-28T11:58:47Z ERROR Timed out trying to obtain keys.
2018-02-28T11:58:47Z ERROR The ipa-replica-install command failed. See /var/log/ipareplica-install.log for more information


and yet:

# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

AND if I add a user on the far end server - ipac1 - it shows up immediately on ipap1.

But, if I try to restart IPAP1 -

# ipactl restart
Upgrade required: please run ipa-server-upgrade command
Aborting ipactl

[root@ipap1 ~]# ipactl status
Directory Service: RUNNING
krb5kdc Service: RUNNING
kadmin Service: RUNNING
httpd Service: RUNNING
ipa-custodia Service: RUNNING
ntpd Service: RUNNING
ipa-otpd Service: RUNNING
ipa: INFO: The ipactl command was successful

So I know something is wrong and I can't leave it this way, but I just don't see what is going on here - can SOMEONE point me in the right direction, please? I don't understand why it won't just rely on IPAP which is the server it is connected to.

thank you,
Kat