We are in the process of adding a new a CA replica.
We install in the following fashion: 1. ipa-replica-install 2. ipa-dns-install 3. ipa-ca-install
All goes well until step3. ipa-ca-install, where we get the error:
2023-05-22T16:51:30Z ERROR ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa011.ad.company.fm"
If we do --skip-conn-check (not recommended) at step 3 we get a complete install, but it does not allow kinit to work on that server.
Any thoughts on how to diagnose and/or fix?
Thanks Nick.
Nicholas Cross via FreeIPA-users wrote:
We are in the process of adding a new a CA replica.
We install in the following fashion:
- ipa-replica-install
- ipa-dns-install
- ipa-ca-install
All goes well until step3. ipa-ca-install, where we get the error:
2023-05-22T16:51:30Z ERROR ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa011.ad.company.fm"
If we do --skip-conn-check (not recommended) at step 3 we get a complete install, but it does not allow kinit to work on that server.
Any thoughts on how to diagnose and/or fix?
We need more context to understand the issue. Can you paste more/all of the replica install log? I don't know where this "must be" message is coming from, nor what service it is complaining about.
What version of IPA and what distribution are the master and replica running?
I don't know how adding a CA would impact running kinit but skipping the connection check often does lead to failures.
rob
My apologies, it has been a long day trying to debug this.
pastebin link to debug logs. https://pastebin.com/U1igJWST
versions replica 4.10.1-6. master 4.10.0-8
# replica [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
#master [root@ipa010 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
Connection check prior to install
[root@ipa011 ~]# ipa-replica-conncheck --master=ipa010.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=cccccccc Check connection from replica to remote master 'ipa010.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Kerberos Kpasswd: TCP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK
The following list of ports use UDP protocol and would need to be checked manually: Kerberos KDC: UDP (88): SKIPPED Kerberos Kpasswd: UDP (464): SKIPPED
Connection from replica to master is OK. Start listening on required ports for remote master check Get credentials to log in to remote master nicholas.cross@AD.companyx.FM password:
Check RPC connection to remote master Could not connect to the remote host: 'Env' object has no attribute 'domain' Retrying using SSH... Check SSH connection to remote master Execute check on remote master Check connection from master to remote replica 'ipa011.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Kerberos KDC: UDP (88): OK Kerberos Kpasswd: TCP (464): OK Kerberos Kpasswd: UDP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK
Connection from master to replica is OK.
Any help would be appreciated.
Nick
Nicholas Cross via FreeIPA-users wrote:
My apologies, it has been a long day trying to debug this.
pastebin link to debug logs. https://pastebin.com/U1igJWST
Is this log a snippet from ipareplica-install.log or from ipa-replica-conncheck?
It looks like the hostname that is being evaluated by conncheck in the call from ipa-replica-install doesn't match the configured host value in /etc/ipa/default.conf (or the system fqdn if that isn't present).
rob
versions replica 4.10.1-6. master 4.10.0-8
# replica [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
#master [root@ipa010 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
Connection check prior to install
[root@ipa011 ~]# ipa-replica-conncheck --master=ipa010.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=cccccccc Check connection from replica to remote master 'ipa010.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Kerberos Kpasswd: TCP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK
The following list of ports use UDP protocol and would need to be checked manually: Kerberos KDC: UDP (88): SKIPPED Kerberos Kpasswd: UDP (464): SKIPPED
Connection from replica to master is OK. Start listening on required ports for remote master check Get credentials to log in to remote master nicholas.cross@AD.companyx.FM password:
Check RPC connection to remote master Could not connect to the remote host: 'Env' object has no attribute 'domain' Retrying using SSH... Check SSH connection to remote master Execute check on remote master Check connection from master to remote replica 'ipa011.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Kerberos KDC: UDP (88): OK Kerberos Kpasswd: TCP (464): OK Kerberos Kpasswd: UDP (464): OK HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK
Connection from master to replica is OK.
Any help would be appreciated.
Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks: - first from the replica to the existing master (here we seem to be good) - then from the existing master to the replica, by doing a call to the XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Flo, Thanks for this. I rebuilt the server again. Did the ipa-replica-install with no DNS or CA install, so just replica. This installed fine. then ran your tests
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM Using existing cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
they look good, but when i kinit as myself... :(
[root@ipa011 ~]# kinit -V nicholas.cross Using new cache: 0:13315 Using principal: nicholas.cross@AD.companyx.FM Password for nicholas.cross@AD.companyx.FM: kinit: Generic error (see e-text) while getting initial credentials
from /var/log/krb5kdc.log on ipa011 the server i am trying to build.
May 23 10:38:01 ipa011.ad.companyx.fm krb5kdc[4303](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17)}) 10.32.225.7: NEEDED_PREAUTH: nicholas.cross@AD.companyx.FM for krbtgt/AD.companyx.FM@AD.companyx.FM, Additional pre-authentication required May 23 10:38:01 ipa011.ad.companyx.fm krb5kdc[4303](info): closing down fd 11 May 23 10:38:03 ipa011.ad.companyx.fm krb5kdc[4303](info): AS_REQ : handle_authdata (2) May 23 10:38:03 ipa011.ad.companyx.fm krb5kdc[4303](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17)}) 10.32.225.7: HANDLE_AUTHDATA: nicholas.cross@AD.companyx.FM for krbtgt/AD.companyx.FM@AD.companyx.FM, No such file or directory May 23 10:38:03 ipa011.ad.companyx.fm krb5kdc[4303](info): closing down fd 11 May 23 10:38:06 ipa011.ad.companyx.fm krb5kdc[4303](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17)}) 10.32.17.117: NEEDED_PREAUTH: host/fullvpn-vpnserver-7fc2c6aba39d946.ad.companyx.fm@AD.companyx.FM for krbtgt/AD.companyx.FM@AD.companyx.FM, Additional pre-authentication required May 23 10:38:06 ipa011.ad.companyx.fm krb5kdc[4303](info): closing down fd 11 May 23 10:38:06 ipa011.ad.companyx.fm krb5kdc[4303](info): preauth (spake) verify failure: Preauthentication failed May 23 10:38:06 ipa011.ad.companyx.fm krb5kdc[4303](info): AS_REQ (4 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), aes128-cts-hmac-sha256-128(19), aes128-cts-hmac-sha1-96(17)}) 10.32.17.117: PREAUTH_FAILED: host/fullvpn-vpnserver-7fc2c6aba39d946.ad.companyx.fm@AD.companyx.FM for krbtgt/AD.companyx.FM@AD.companyx.FM, Preauthentication failed May 23 10:38:06 ipa011.ad.companyx.fm krb5kdc[4303](info): closing down fd 11
I hope this makes some sense? :)
Hi Flo (and other helpful people on this list),
After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.
I ran your suggested commands and they look fine now.
[root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ ipa011.ad.companyx.fm@AD.companyx.FM Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.
ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.
Some logs: 1. before ipa-replica-install - conn check works. http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle
new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
thanks, Nick
On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be good)
- then from the existing master to the replica, by doing a call to the
XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
On Wed, May 24, 2023 at 3:29 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Flo (and other helpful people on this list),
After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.
I ran your suggested commands and they look fine now.
[root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ ipa011.ad.companyx.fm@AD.companyx.FM Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.
ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.
ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation
The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc...
I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here.
flo
Some logs:
- before ipa-replica-install - conn check works.
http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle
new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
thanks, Nick
On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be good)
- then from the existing master to the replica, by doing a call to the
XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi, thanks for the reply.
This was the process. 1. prep 2. test connectivity - works ok 3. install replica (successfully) - no DNS no CA at this point 4. test connectivity - fails.
If i replace step 4 for the following:
4. install DNS (successfully) 5. install CA (fails - with the "invalid 'cn': must be" error)
If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL)
Successful DNS install logs http://sprunge.us/7WCtpW
Failed CA install logs http://sprunge.us/hTVLzL
[root@ipa011 ~]# ipa-ca-install Directory Manager (existing master) password:
Run connection check to master
Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter.
thanks, Nick
On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
On Wed, May 24, 2023 at 3:29 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Flo (and other helpful people on this list),
After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.
I ran your suggested commands and they look fine now.
[root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ ipa011.ad.companyx.fm@AD.companyx.FM Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.
ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.
ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation
The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc...
I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here.
flo
Some logs:
- before ipa-replica-install - conn check works.
http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle
new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
thanks, Nick
On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be good)
- then from the existing master to the replica, by doing a call to the
XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi, thanks for the reply.
This was the process.
- prep
- test connectivity - works ok
- install replica (successfully) - no DNS no CA at this point
- test connectivity - fails.
If i replace step 4 for the following:
- install DNS (successfully)
- install CA (fails - with the "invalid 'cn': must be" error)
If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL)
Successful DNS install logs http://sprunge.us/7WCtpW
Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure: - create /etc/ipa/server.conf: [global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
flo
[root@ipa011 ~]# ipa-ca-install
Directory Manager (existing master) password:
Run connection check to master
Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter.
thanks, Nick
On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
On Wed, May 24, 2023 at 3:29 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Flo (and other helpful people on this list),
After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.
I ran your suggested commands and they look fine now.
[root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ ipa011.ad.companyx.fm@AD.companyx.FM Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.
ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.
ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation
The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc...
I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here.
flo
Some logs:
- before ipa-replica-install - conn check works.
http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle
new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
thanks, Nick
On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be
good)
- then from the existing master to the replica, by doing a call to the
XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
I turned on the debugging, but it showed nothing at all when i ran the conn-check command again. :(
Interestingly this turn up in the http errors log when restarted it (but before i ran conn-check)
[Thu May 25 09:58:25.521316 2023] [wsgi:error] [pid 2226275:tid 2226275] ipa: ERROR: Failed to pre-populate LDAP schema cache: uri=ldapi://%2Frun%2Fslapd-AD-companyx-FM.socket: Unable to retrieve LDAP schema: Inappropriate authentication: Anonymous access is not allowed.
Is this just a warning? We do turn off full anonymous access for Security reasons.
Also, when running conn-check, I get the same error if i run the conn-check command against my new server ipa011 from any other current IPA server eg. from ipa008 to ipa011, or ipa004 to ipa011. But not if i run from a current server to another current server. eg. ipa008 to ipa004 this works fine. So it seems the issue is just this newly installed IPA server.
# # from ipa011 to ipa008 #
Check RPC connection to remote master Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa004dc.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Connection to https://ipa006dc.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Execute check on remote master ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa011.ad.companyx.fm" timed out waiting for input: auto-logout
# # from ipa008 to ipa011 # Check RPC connection to remote master Connection to https://ipa011.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Execute check on remote master ERROR: Remote master check failed with following error message(s): invalid 'cn': must be "ipa008.ad.companyx.fm"
# # from ipa008 to ipa010 # Check RPC connection to remote master Execute check on remote master Check connection from master to remote replica 'ipa008.ad.companyx.fm': Directory Service: Unsecure port (389): OK Directory Service: Secure port (636): OK Kerberos KDC: TCP (88): OK Failed to connect to port 88 udp on 10.32.218.203 Kerberos KDC: UDP (88): WARNING Kerberos Kpasswd: TCP (464): OK Failed to connect to port 464 udp on 10.32.218.203 Kerberos Kpasswd: UDP (464): WARNING HTTP Server: Unsecure port (80): OK HTTP Server: Secure port (443): OK The following UDP ports could not be verified as open: 88, 464 This can happen if they are already bound to an application and ipa-replica-conncheck cannot attach own UDP responder.
Connection from master to replica is OK.
Currently, i am trying to find the python code that spits out the error and then maybe to tweak it a little to throw some more debug out.
maybe here? https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L...
thanks, Nick
On Thu, 25 May 2023 at 10:13, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi, thanks for the reply.
This was the process.
- prep
- test connectivity - works ok
- install replica (successfully) - no DNS no CA at this point
- test connectivity - fails.
If i replace step 4 for the following:
- install DNS (successfully)
- install CA (fails - with the "invalid 'cn': must be" error)
If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL)
Successful DNS install logs http://sprunge.us/7WCtpW
Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
flo
[root@ipa011 ~]# ipa-ca-install
Directory Manager (existing master) password:
Run connection check to master
Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up.
Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter.
thanks, Nick
On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
On Wed, May 24, 2023 at 3:29 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Flo (and other helpful people on this list),
After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.
I ran your suggested commands and they look fine now.
[root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache
[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ ipa011.ad.companyx.fm@AD.companyx.FM Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5
[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1
I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.
ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.
ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation
The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc...
I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here.
flo
Some logs:
- before ipa-replica-install - conn check works.
http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle
new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64
current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64
thanks, Nick
On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be
good)
- then from the existing master to the replica, by doing a call to the
XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo
On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users < freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file
it does looks like a DNs issue, but im not sure where.
dns resolves the host fine on the host
[root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm has address 10.32.225.7
[root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm
it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error.
As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference.
We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them.
I will update the client to ignore the forwarders, but if you can think of anything else to try?
thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm mailto:nicholas.cross@dice.fm> wrote:
Hi, thanks for the reply. This was the process. 1. prep 2. test connectivity - works ok 3. install replica (successfully) - no DNS no CA at this point 4. test connectivity - fails. If i replace step 4 for the following: 4. install DNS (successfully) 5. install CA (fails - with the "invalid 'cn': must be" error) If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL) Successful DNS install logs http://sprunge.us/7WCtpW Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
It would also be interesting to see the entries around the 'server_conncheck' call in /var/log/httpd/error_log to see what arguments were passed in (on ipa011) since you're enabling debug mode anyway.
rob
flo
[root@ipa011 ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter. thanks, Nick On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>> wrote: Hi Flo (and other helpful people on this list), After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica. I ran your suggested commands and they look fine now. [root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5 [root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1 I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work. ipa-replica-conncheck --master=ipa008.ad.companyx.fm <http://ipa008.ad.companyx.fm> --auto-master-check --realm=AD.companyx.FM <http://AD.companyx.FM> --principal=nicholas.cross After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described. ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc... I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here. flo Some logs: 1. before ipa-replica-install - conn check works. http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check fails http://sprunge.us/gvqEle new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64 current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64 thanks, Nick On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks: - first from the replica to the existing master (here we seem to be good) - then from the existing master to the replica, by doing a call to the XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> wrote: That was the /var/log/ipareplica-conncheck.log log file it does looks like a DNs issue, but im not sure where. dns resolves the host fine on the host [root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> has address 10.32.225.7 [root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm <http://ipa010.ad.companyx.fm> it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error. As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference. We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them. I will update the client to ignore the forwarders, but if you can think of anything else to try? thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi Rob,
Thanks for the tip, it came in handy with what i was working on (the python code).
Here is the error_log on the new server (ipa011)
[Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping(): SUCCESS [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping/1(version='2.251'): SUCCESS [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm', ' ipa011.ad.companyx.fm', version='2.162'): ValidationError
I tweaked the code a little to do some old skool debugging... here: https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L...
This enabled me to dump the vars in question to why the error was raised:
ERROR: Remote master check failed with following error message(s): invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:(' ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]: ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
So: keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
As the code states:
if keys[-2] != api.env.host: raise errors.ValidationError( name='cn', error=_("must be "%s"") % api.env.host)
and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fm
So, that's about as far as i have gotten so far.
Do we think the keys swapped around for some reason?
thanks, Nick
On Thu, 25 May 2023 at 13:52, Rob Crittenden rcritten@redhat.com wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm mailto:nicholas.cross@dice.fm> wrote:
Hi, thanks for the reply. This was the process. 1. prep 2. test connectivity - works ok 3. install replica (successfully) - no DNS no CA at this point 4. test connectivity - fails. If i replace step 4 for the following: 4. install DNS (successfully) 5. install CA (fails - with the "invalid 'cn': must be" error) If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL) Successful DNS install logs http://sprunge.us/7WCtpW Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to
https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
It would also be interesting to see the entries around the 'server_conncheck' call in /var/log/httpd/error_log to see what arguments were passed in (on ipa011) since you're enabling debug mode anyway.
rob
flo
[root@ipa011 ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter. thanks, Nick On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>> wrote: Hi Flo (and other helpful people on this list), After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica. I ran your suggested commands and they look fine now. [root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5 [root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1 I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work. ipa-replica-conncheck --master=ipa008.ad.companyx.fm <http://ipa008.ad.companyx.fm> --auto-master-check --realm=AD.companyx.FM <http://AD.companyx.FM> --principal=nicholas.cross After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described. ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc... I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here. flo Some logs: 1. before ipa-replica-install - conn check works. http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check
fails
http://sprunge.us/gvqEle new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64 current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64 thanks, Nick On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks: - first from the replica to the existing master (here we seem to be good) - then from the existing master to the replica, by doing a call to the XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is
empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> wrote: That was the /var/log/ipareplica-conncheck.log log
file
it does looks like a DNs issue, but im not sure
where.
dns resolves the host fine on the host [root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> has address 10.32.225.7 [root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm <http://ipa010.ad.companyx.fm> it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error. As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference. We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them. I will update the client to ignore the forwarders, but if you can think of anything else to try? thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it:
Hi,
in your logs we see the following:
2023-05-25T07:52:07Z DEBUG trying https://ipa008.ad.companyx.fm/ipa/json 2023-05-25T07:52:07Z DEBUG New HTTP connection (ipa008.ad.companyx.fm) 2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']' 2023-05-25T07:52:07Z DEBUG storing cookie 'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;' for principal host/ipa011.ad.companyx.fm@AD.companyx.FM 2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) 2023-05-25T07:52:07Z DEBUG trying https://ipa011.ad.companyx.fm/ipa/json
This shows that in a first pass, the replica tries to connect to the master ipa008 in order to do RPC calls. This corresponds to this piece of code https://github.com/freeipa/freeipa/blob/master/install/tools/ipa-replica-con... where the api.Backend.rpcclient.connect() method internally tries to reach one of the IPA servers (for the details, see https://github.com/freeipa/freeipa/blob/master/ipalib/rpc.py#L1002, the available servers are tried sequentially in the *for url in urls:* loop). In your case, it fails on ipa008 and continues with ipa011 (the replica). The call to the api server_conncheck is then executed on the local replica and this explains the error: we don't want to check connection from local to local, we want to check from ipa008 to ipa011.
The piece of code that needs investigation is why does the connection to ipa008 fail with GSSAPI error. The debug must be set on the existing master, in order to see why the authentication fails there.
HTH, flo
On Thu, May 25, 2023 at 3:56 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Rob,
Thanks for the tip, it came in handy with what i was working on (the python code).
Here is the error_log on the new server (ipa011)
[Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping(): SUCCESS [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping/1(version='2.251'): SUCCESS [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm', ' ipa011.ad.companyx.fm', version='2.162'): ValidationError
I tweaked the code a little to do some old skool debugging... here: https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L...
This enabled me to dump the vars in question to why the error was raised:
ERROR: Remote master check failed with following error message(s): invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:(' ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]: ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
So: keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
As the code states:
if keys[-2] != api.env.host: raise errors.ValidationError( name='cn', error=_("must be "%s"") % api.env.host)
and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fm
So, that's about as far as i have gotten so far.
Do we think the keys swapped around for some reason?
thanks, Nick
On Thu, 25 May 2023 at 13:52, Rob Crittenden rcritten@redhat.com wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm mailto:nicholas.cross@dice.fm> wrote:
Hi, thanks for the reply. This was the process. 1. prep 2. test connectivity - works ok 3. install replica (successfully) - no DNS no CA at this point 4. test connectivity - fails. If i replace step 4 for the following: 4. install DNS (successfully) 5. install CA (fails - with the "invalid 'cn': must be" error) If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL) Successful DNS install logs http://sprunge.us/7WCtpW Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to
https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
It would also be interesting to see the entries around the 'server_conncheck' call in /var/log/httpd/error_log to see what arguments were passed in (on ipa011) since you're enabling debug mode anyway.
rob
flo
[root@ipa011 ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter. thanks, Nick On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>> wrote: Hi Flo (and other helpful people on this list), After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica. I ran your suggested commands and they look fine now. [root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5 [root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1 I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work. ipa-replica-conncheck --master=ipa008.ad.companyx.fm <http://ipa008.ad.companyx.fm> --auto-master-check --realm=AD.companyx.FM <http://AD.companyx.FM> --principal=nicholas.cross After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described. ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc... I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here. flo Some logs: 1. before ipa-replica-install - conn check works. http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check
fails
http://sprunge.us/gvqEle new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64 current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64 thanks, Nick On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks: - first from the replica to the existing master (here we seem to be good) - then from the existing master to the replica, by doing a call to the XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of SASL auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is
empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ? flo On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> wrote: That was the /var/log/ipareplica-conncheck.log log
file
it does looks like a DNs issue, but im not sure
where.
dns resolves the host fine on the host [root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm has address 10.32.225.7 [root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm <http://ipa010.ad.companyx.fm> it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error. As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference. We do have new DNS forwarders on the network - these are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them. I will update the client to ignore the forwarders, but if you can think of anything else to try? thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org> Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it:
So, some good news.
I tried this again today and it works, i turned on the httpd debugging as suggested on the master and it worked. O_O
The only thing i have changed outside of the work we were doing here was that i noticed we had a lot of ghost PKI servers yesterday in `pki securitydomain-show` So i followed https://access.redhat.com/solutions/6615411 and removed them.
If this fixed it i don't know, how it could affect it again i dont know, but it was the only other change we made.
I shall test again on Tuesday to see if we are still ok.
thanks for your continued help on this. have a great weekend. Nick.
On Fri, 26 May 2023 at 09:35, Florence Blanc-Renaud flo@redhat.com wrote:
Hi,
in your logs we see the following:
2023-05-25T07:52:07Z DEBUG trying https://ipa008.ad.companyx.fm/ipa/json 2023-05-25T07:52:07Z https://ipa008.ad.companyx.fm/ipa/json%0D2023-05-25T07:52:07Z DEBUG New HTTP connection (ipa008.ad.companyx.fm) 2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']' 2023-05-25T07:52:07Z DEBUG storing cookie 'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;' for principal host/ipa011.ad.companyx.fm@AD.companyx.FM 2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) 2023-05-25T07:52:07Z DEBUG trying https://ipa011.ad.companyx.fm/ipa/json
This shows that in a first pass, the replica tries to connect to the master ipa008 in order to do RPC calls. This corresponds to this piece of code https://github.com/freeipa/freeipa/blob/master/install/tools/ipa-replica-con... where the api.Backend.rpcclient.connect() method internally tries to reach one of the IPA servers (for the details, see https://github.com/freeipa/freeipa/blob/master/ipalib/rpc.py#L1002, the available servers are tried sequentially in the *for url in urls:* loop). In your case, it fails on ipa008 and continues with ipa011 (the replica). The call to the api server_conncheck is then executed on the local replica and this explains the error: we don't want to check connection from local to local, we want to check from ipa008 to ipa011.
The piece of code that needs investigation is why does the connection to ipa008 fail with GSSAPI error. The debug must be set on the existing master, in order to see why the authentication fails there.
HTH, flo
On Thu, May 25, 2023 at 3:56 PM Nicholas Cross nicholas.cross@dice.fm wrote:
Hi Rob,
Thanks for the tip, it came in handy with what i was working on (the python code).
Here is the error_log on the new server (ipa011)
[Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec) [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND' [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping(): SUCCESS [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping/1(version='2.251'): SUCCESS [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm', version='2.162'): ValidationError
I tweaked the code a little to do some old skool debugging... here: https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.py#L...
This enabled me to dump the vars in question to why the error was raised:
ERROR: Remote master check failed with following error message(s): invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:(' ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]: ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
So: keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
As the code states:
if keys[-2] != api.env.host: raise errors.ValidationError( name='cn', error=_("must be "%s"") % api.env.host)
and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fm
So, that's about as far as i have gotten so far.
Do we think the keys swapped around for some reason?
thanks, Nick
On Thu, 25 May 2023 at 13:52, Rob Crittenden rcritten@redhat.com wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm mailto:nicholas.cross@dice.fm> wrote:
Hi, thanks for the reply. This was the process. 1. prep 2. test connectivity - works ok 3. install replica (successfully) - no DNS no CA at this point 4. test connectivity - fails. If i replace step 4 for the following: 4. install DNS (successfully) 5. install CA (fails - with the "invalid 'cn': must be" error) If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so. But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL) Successful DNS install logs http://sprunge.us/7WCtpW Failed CA install logs http://sprunge.us/hTVLzL
I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to
https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global] debug=True
then restart apache with systemctl restart httpd. You can also run ipa-healthcheck on the master/the replica to find any config issue.
It would also be interesting to see the entries around the 'server_conncheck' call in /var/log/httpd/error_log to see what arguments were passed in (on ipa011) since you're enabling debug mode anyway.
rob
flo
[root@ipa011 ~]# ipa-ca-install Directory Manager (existing master) password: Run connection check to master Your system may be partly configured. Run /usr/sbin/ipa-server-install --uninstall to clean up. Connection check failed! See /var/log/ipareplica-conncheck.log for more information. If the check results are not valid it can be skipped with --skip-conncheck parameter. thanks, Nick On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <
flo@redhat.com
<mailto:flo@redhat.com>> wrote: Hi, On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>>
wrote:
Hi Flo (and other helpful people on this list), After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica. I ran your suggested commands and they look fine now. [root@ipa011 ~]# kdestroy kdestroy: No credentials cache found while destroying cache [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using default cache: 0 Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> Using keytab: /etc/krb5.keytab Authenticated to Kerberos v5 [root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1 I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work. ipa-replica-conncheck --master=ipa008.ad.companyx.fm <http://ipa008.ad.companyx.fm> --auto-master-check --realm=AD.companyx.FM <http://AD.companyx.FM> --principal=nicholas.cross After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described. ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page: ipa-replica-conncheck - Check a replica-master network connection before installation The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If
the
replica is already installed, the ports are already in use by the ldap server, the apache web server etc... I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here. flo Some logs: 1. before ipa-replica-install - conn check works. http://sprunge.us/4yrInD 2. after successful ipa-replica-install - but conn check
fails
http://sprunge.us/gvqEle new replica versions [root@ipa011 ~]# rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.1-6.el9.x86_64 ipa-client-common-4.10.1-6.el9.noarch ipa-common-4.10.1-6.el9.noarch ipa-healthcheck-0.12-1.el9.noarch ipa-healthcheck-core-0.12-1.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.1-6.el9.x86_64 ipa-server-common-4.10.1-6.el9.noarch ipa-server-dns-4.10.1-6.el9.noarch libipa_hbac-2.8.2-2.el9.x86_64 python3-ipaclient-4.10.1-6.el9.noarch python3-ipalib-4.10.1-6.el9.noarch python3-ipaserver-4.10.1-6.el9.noarch python3-libipa_hbac-2.8.2-2.el9.x86_64 sssd-ipa-2.8.2-2.el9.x86_64 current master versions [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort almalinux-logos-ipa-90.5.1-1.1.el9.noarch ipa-client-4.10.0-8.el9_1.x86_64 ipa-client-common-4.10.0-8.el9_1.noarch ipa-common-4.10.0-8.el9_1.noarch ipa-healthcheck-0.9-9.el9.noarch ipa-healthcheck-core-0.9-9.el9.noarch ipa-selinux-4.9.8-7.el9_0.noarch ipa-server-4.10.0-8.el9_1.x86_64 ipa-server-common-4.10.0-8.el9_1.noarch ipa-server-dns-4.10.0-8.el9_1.noarch libipa_hbac-2.7.3-4.el9_1.3.x86_64 python3-ipaclient-4.10.0-8.el9_1.noarch python3-ipalib-4.10.0-8.el9_1.noarch python3-ipaserver-4.10.0-8.el9_1.noarch python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64 sssd-ipa-2.7.3-4.el9_1.3.x86_64 thanks, Nick On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo@redhat.com <mailto:flo@redhat.com>> wrote: Hi, the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks: - first from the replica to the existing master (here
we
seem to be good) - then from the existing master to the replica, by
doing
a call to the XMLRPC api server_conncheck on the
master.
If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call. The logs shows that the connection fails because of
SASL
auth failure: 2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is
empty)
Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then
kvno
HTTP/<serverfqdn>@<REALM> ? flo On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org>> wrote: That was the /var/log/ipareplica-conncheck.log log
file
it does looks like a DNs issue, but im not sure
where.
dns resolves the host fine on the host [root@ipa011 ~]# host ipa011 ipa011.ad.companyx.fm <
http://ipa011.ad.companyx.fm%3E
has address 10.32.225.7 [root@ipa011 ~]# grep ipa /etc/ipa/default.conf host = ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml ca_host = ipa010.ad.companyx.fm <http://ipa010.ad.companyx.fm> it's odd as i run the connection check before the start of the install, to check ports and routes. it works fine. replica install works. dns install works. just the ca installer comes back with this error. As an additional test i added the dns record for this host into IPA before the install. Normally we don't need to, but just as a test, but it made no difference. We do have new DNS forwarders on the network -
these
are in front of the IPA servers. They are there just take the load from the k8s clusters away from IPA DNS. Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders? All DNS tests i can think of work via the forwarders. The IPA clients (100s) are all fine with them. I will update the client to ignore the forwarders, but if you can think of anything else to try? thanks, Nick _______________________________________________ FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org <mailto:freeipa-users@lists.fedorahosted.org> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org <mailto:freeipa-users-leave@lists.fedorahosted.org Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org To unsubscribe send an email to
freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahoste...
Do not reply to spam, report it:
freeipa-users@lists.fedorahosted.org