Hi,

On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm> wrote:
Hi, thanks for the reply.

This was the process.
1. prep
2. test connectivity - works ok
3. install replica (successfully) - no DNS no CA at this point
4. test connectivity - fails.

If i replace step 4 for the following:

4. install DNS (successfully)
5. install CA (fails - with the "invalid 'cn': must be" error)

If we can ignore the step 4 where the comms check fails and concentrate on step 5 where the CA fails. I am happy to do so.
But in step 5. it is the comms check that the `ipa-ca-install` command runs that also fails as far as i can see (no expert here myself, though my company thinks so LOL)

Successful DNS install logs http://sprunge.us/7WCtpW 

Failed CA install logs  http://sprunge.us/hTVLzL

I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a cookie but later on GSSAPI auth fails.

Are there any corresponding logs in the master in /var/log/httpd/error_log? You can enable debug logs on the master with the following procedure:
- create /etc/ipa/server.conf:
[global]
debug=True

then restart apache with systemctl restart httpd.
You can also run ipa-healthcheck on the master/the replica to find any config issue.

flo

[root@ipa011 ~]# ipa-ca-install
Directory Manager (existing master) password:

Run connection check to master

Your system may be partly configured.
Run /usr/sbin/ipa-server-install --uninstall to clean up.

Connection check failed!
See /var/log/ipareplica-conncheck.log for more information.
If the check results are not valid it can be skipped with --skip-conncheck parameter.



thanks,
Nick

On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com> wrote:
Hi,

On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross@dice.fm> wrote:
Hi Flo (and other helpful people on this list),

After fixing the SID/PAC issue, i am back to looking as to why the ipa-replica-conncheck fails when installing the CA to a (working) replica.

I ran your suggested commands and they look fine now.

[root@ipa011 ~]# kdestroy
kdestroy: No credentials cache found while destroying cache

[root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/ipa011.ad.companyx.fm@AD.companyx.FM
Using default cache: 0
Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM
Using keytab: /etc/krb5.keytab
Authenticated to Kerberos v5

[root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm@AD.companyx.FM
HTTP/ipa011.ad.companyx.fm@AD.companyx.FM: kvno = 1

I can run the following BEFORE i install the replica and it works 100%, it is part of our pre-reqs in our docs to make sure comms work.

ipa-replica-conncheck --master=ipa008.ad.companyx.fm --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross

After i install the replica with ipa-replica-install. (no DNS, no CA) then running the above ipa-replica-conncheck command fails and we see the errors as described.

ipa-replica-conncheck is intended to be run *before* replica installation, as stated in the man page:
       ipa-replica-conncheck - Check a replica-master network connection before installation

The reason is that it opens server sockets listening on the tested ports, for the master to try to reach the replica. If the replica is already installed, the ports are already in use by the ldap server, the apache web server etc...

I understand that you tried to run ipa-replica-conncheck to ensure that ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try to install the DNS service and the CA service on the replica? If you see errors in those steps we can help troubleshoot, based on the produced logs, but ipa-replica-conncheck won't be of any use here.

flo

Some logs:
1. before ipa-replica-install - conn check works.
2. after successful ipa-replica-install - but conn check fails

new replica versions
[root@ipa011 ~]# rpm -qa | grep ipa | sort
almalinux-logos-ipa-90.5.1-1.1.el9.noarch
ipa-client-4.10.1-6.el9.x86_64
ipa-client-common-4.10.1-6.el9.noarch
ipa-common-4.10.1-6.el9.noarch
ipa-healthcheck-0.12-1.el9.noarch
ipa-healthcheck-core-0.12-1.el9.noarch
ipa-selinux-4.9.8-7.el9_0.noarch
ipa-server-4.10.1-6.el9.x86_64
ipa-server-common-4.10.1-6.el9.noarch
ipa-server-dns-4.10.1-6.el9.noarch
libipa_hbac-2.8.2-2.el9.x86_64
python3-ipaclient-4.10.1-6.el9.noarch
python3-ipalib-4.10.1-6.el9.noarch
python3-ipaserver-4.10.1-6.el9.noarch
python3-libipa_hbac-2.8.2-2.el9.x86_64
sssd-ipa-2.8.2-2.el9.x86_64

current master versions
[nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort
almalinux-logos-ipa-90.5.1-1.1.el9.noarch
ipa-client-4.10.0-8.el9_1.x86_64
ipa-client-common-4.10.0-8.el9_1.noarch
ipa-common-4.10.0-8.el9_1.noarch
ipa-healthcheck-0.9-9.el9.noarch
ipa-healthcheck-core-0.9-9.el9.noarch
ipa-selinux-4.9.8-7.el9_0.noarch
ipa-server-4.10.0-8.el9_1.x86_64
ipa-server-common-4.10.0-8.el9_1.noarch
ipa-server-dns-4.10.0-8.el9_1.noarch
libipa_hbac-2.7.3-4.el9_1.3.x86_64
python3-ipaclient-4.10.0-8.el9_1.noarch
python3-ipalib-4.10.0-8.el9_1.noarch
python3-ipaserver-4.10.0-8.el9_1.noarch
python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
sssd-ipa-2.7.3-4.el9_1.3.x86_64

thanks,
Nick

On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo@redhat.com> wrote:
Hi,

the replica-conncheck error means that a call to server_conncheck reached the wrong server. ipa-replica-conncheck performs multiple checks:
- first from the replica to the existing master (here we seem to be good)
- then from the existing master to the replica, by doing a call to the XMLRPC api server_conncheck on the master. If the connection from replica to master fails, another server is tried (in this case, the replica launches server_conncheck on itself), but there is a security that ensures the right server is handling the call.
The logs shows that the connection fails because of SASL auth failure:
2023-05-22T18:14:03Z INFO Connection to https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)

Are you able to do kinit -kt /etc/krb5.keytab host/<replicafqdn>@<REALM> on the replica? And then kvno HTTP/<serverfqdn>@<REALM> ?
flo

On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <freeipa-users@lists.fedorahosted.org> wrote:
That was the /var/log/ipareplica-conncheck.log log file

it does looks like a DNs issue, but im not sure where.

dns resolves the host fine on the host

[root@ipa011 ~]# host ipa011
ipa011.ad.companyx.fm has address 10.32.225.7

[root@ipa011 ~]# grep ipa /etc/ipa/default.conf
host = ipa011.ad.companyx.fm
xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml
ca_host = ipa010.ad.companyx.fm

it's odd as i run the connection check before the start of the install, to check ports and routes.   it works fine.
replica install works.
dns install works.
just the ca installer comes back with this error.

As an additional test i added the dns record for this host into IPA before the install.  Normally we don't need to, but just as a test, but it made no difference.


We do have new DNS forwarders on the network - these are in front of the IPA servers.   They are there just take the load from the k8s clusters away from IPA DNS.
Would the CA install break if the DNS lookups are "proxied" by the DNS forwarders?
All DNS tests i can think of work via the forwarders.  The IPA clients (100s) are all fine with them.

I will update the client to ignore the forwarders, but if you can think of anything else to try?

thanks, Nick
_______________________________________________
FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue