2023-05-25T07:52:07Z DEBUG trying https://ipa008.ad.companyx.fm/ipa/json 2023-05-25T07:52:07Z DEBUG New HTTP connection (ipa008.ad.companyx.fm) 2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']' 2023-05-25T07:52:07Z DEBUG storing cookie 'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;' for principal host/ipa011.ad.companyx.fm@AD.companyx.FM 2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty) 2023-05-25T07:52:07Z DEBUG trying https://ipa011.ad.companyx.fm/ipa/json
Hi Rob,Thanks for the tip, it came in handy with what i was working on (the python code).Here is the error_log on the new server (ipa011)[Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully
[Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor
[Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations
[Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping(): SUCCESS
[Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping/1(version='2.251'): SUCCESS
[Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm', version='2.162'): ValidationErrorI tweaked the code a little to do some old skool debugging...This enabled me to dump the vars in question to why the error was raised:
ERROR: Remote master check failed with following error message(s):
invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
So:
keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm')keys[-2]:ipa008.ad.companyx.fmAs the code states:if keys[-2] != api.env.host:
raise errors.ValidationError(
name='cn', error=_("must be \"%s\"") % api.env.host)and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fmSo, that's about as far as i have gotten so far.Do we think the keys swapped around for some reason?thanks,NickOn Thu, 25 May 2023 at 13:52, Rob Crittenden <rcritten@redhat.com> wrote:Florence Blanc-Renaud via FreeIPA-users wrote:
> Hi,
>
> On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm
> <mailto:nicholas.cross@dice.fm>> wrote:
>
> Hi, thanks for the reply.
>
> This was the process.
> 1. prep
> 2. test connectivity - works ok
> 3. install replica (successfully) - no DNS no CA at this point
> 4. test connectivity - fails.
>
> If i replace step 4 for the following:
>
> 4. install DNS (successfully)
> 5. install CA (fails - with the "invalid 'cn': must be" error)
>
> If we can ignore the step 4 where the comms check fails and
> concentrate on step 5 where the CA fails. I am happy to do so.
> But in step 5. it is the comms check that the `ipa-ca-install`
> command runs that also fails as far as i can see (no expert here
> myself, though my company thinks so LOL)
>
> Successful DNS install logs http://sprunge.us/7WCtpW
>
> Failed CA install logs http://sprunge.us/hTVLzL
>
> I believe this error needs to be investigated:
>
> 2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
>
> We can see that the process is able to connect to the master and get a
> cookie but later on GSSAPI auth fails.
>
> Are there any corresponding logs in the master in
> /var/log/httpd/error_log? You can enable debug logs on the master with
> the following procedure:
> - create /etc/ipa/server.conf:
> [global]
> debug=True
>
> then restart apache with systemctl restart httpd.
> You can also run ipa-healthcheck on the master/the replica to find any
> config issue.
It would also be interesting to see the entries around the
'server_conncheck' call in /var/log/httpd/error_log to see what
arguments were passed in (on ipa011) since you're enabling debug mode
anyway.
rob
>
> flo
>
> [root@ipa011 ~]# ipa-ca-install
> Directory Manager (existing master) password:
>
> Run connection check to master
>
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> Connection check failed!
> See /var/log/ipareplica-conncheck.log for more information.
> If the check results are not valid it can be skipped with
> --skip-conncheck parameter.
>
>
>
> thanks,
> Nick
>
> On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com
> <mailto:flo@redhat.com>> wrote:
>
> Hi,
>
> On Wed, May 24, 2023 at 3:29 PM Nicholas Cross
> <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>> wrote:
>
> Hi Flo (and other helpful people on this list),
>
> After fixing the SID/PAC issue, i am back to looking as to
> why the ipa-replica-conncheck fails when installing the CA
> to a (working) replica.
>
> I ran your suggested commands and they look fine now.
>
> [root@ipa011 ~]# kdestroy
> kdestroy: No credentials cache found while destroying cache
>
> [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab
> host/ipa011.ad.companyx.fm@AD.companyx.FM
> <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
> Using default cache: 0
> Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM
> <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
> Using keytab: /etc/krb5.keytab
> Authenticated to Kerberos v5
>
> [root@ipa011 ~]# kvno
> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM
> <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
> HTTP/ipa011.ad.companyx.fm@AD.companyx.FM
> <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1
>
> I can run the following BEFORE i install the replica and it
> works 100%, it is part of our pre-reqs in our docs to make
> sure comms work.
>
> ipa-replica-conncheck --master=ipa008.ad.companyx.fm
> <http://ipa008.ad.companyx.fm> --auto-master-check
> --realm=AD.companyx.FM <http://AD.companyx.FM>
> --principal=nicholas.cross
>
> After i install the replica with ipa-replica-install. (no
> DNS, no CA) then running the above ipa-replica-conncheck
> command fails and we see the errors as described.
>
>
> ipa-replica-conncheck is intended to be run *before* replica
> installation, as stated in the man page:
> ipa-replica-conncheck - Check a replica-master network
> connection before installation
>
> The reason is that it opens server sockets listening on the
> tested ports, for the master to try to reach the replica. If the
> replica is already installed, the ports are already in use by
> the ldap server, the apache web server etc...
>
> I understand that you tried to run ipa-replica-conncheck to
> ensure that ipa-dns-install or ipa-ca-install won't hit any
> issue. Did you actually try to install the DNS service and the
> CA service on the replica? If you see errors in those steps we
> can help troubleshoot, based on the produced logs, but
> ipa-replica-conncheck won't be of any use here.
>
> flo
>
>
> Some logs:
> 1. before ipa-replica-install - conn check works.
> http://sprunge.us/4yrInD
> 2. after successful ipa-replica-install - but conn check fails
> http://sprunge.us/gvqEle
>
> new replica versions
> [root@ipa011 ~]# rpm -qa | grep ipa | sort
> almalinux-logos-ipa-90.5.1-1.1.el9.noarch
> ipa-client-4.10.1-6.el9.x86_64
> ipa-client-common-4.10.1-6.el9.noarch
> ipa-common-4.10.1-6.el9.noarch
> ipa-healthcheck-0.12-1.el9.noarch
> ipa-healthcheck-core-0.12-1.el9.noarch
> ipa-selinux-4.9.8-7.el9_0.noarch
> ipa-server-4.10.1-6.el9.x86_64
> ipa-server-common-4.10.1-6.el9.noarch
> ipa-server-dns-4.10.1-6.el9.noarch
> libipa_hbac-2.8.2-2.el9.x86_64
> python3-ipaclient-4.10.1-6.el9.noarch
> python3-ipalib-4.10.1-6.el9.noarch
> python3-ipaserver-4.10.1-6.el9.noarch
> python3-libipa_hbac-2.8.2-2.el9.x86_64
> sssd-ipa-2.8.2-2.el9.x86_64
>
> current master versions
> [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort
> almalinux-logos-ipa-90.5.1-1.1.el9.noarch
> ipa-client-4.10.0-8.el9_1.x86_64
> ipa-client-common-4.10.0-8.el9_1.noarch
> ipa-common-4.10.0-8.el9_1.noarch
> ipa-healthcheck-0.9-9.el9.noarch
> ipa-healthcheck-core-0.9-9.el9.noarch
> ipa-selinux-4.9.8-7.el9_0.noarch
> ipa-server-4.10.0-8.el9_1.x86_64
> ipa-server-common-4.10.0-8.el9_1.noarch
> ipa-server-dns-4.10.0-8.el9_1.noarch
> libipa_hbac-2.7.3-4.el9_1.3.x86_64
> python3-ipaclient-4.10.0-8.el9_1.noarch
> python3-ipalib-4.10.0-8.el9_1.noarch
> python3-ipaserver-4.10.0-8.el9_1.noarch
> python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
> sssd-ipa-2.7.3-4.el9_1.3.x86_64
>
> thanks,
> Nick
>
> On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud
> <flo@redhat.com <mailto:flo@redhat.com>> wrote:
>
> Hi,
>
> the replica-conncheck error means that a call to
> server_conncheck reached the wrong server.
> ipa-replica-conncheck performs multiple checks:
> - first from the replica to the existing master (here we
> seem to be good)
> - then from the existing master to the replica, by doing
> a call to the XMLRPC api server_conncheck on the master.
> If the connection from replica to master fails, another
> server is tried (in this case, the replica launches
> server_conncheck on itself), but there is a security
> that ensures the right server is handling the call.
> The logs shows that the connection fails because of SASL
> auth failure:
> 2023-05-22T18:14:03Z INFO Connection to
> https://ipa010.ad.companyx.fm/ipa/json failed with
> Insufficient access: SASL(-1): generic failure: GSSAPI
> Error: No credentials were supplied, or the credentials
> were unavailable or inaccessible (Credential cache is empty)
>
> Are you able to do kinit -kt /etc/krb5.keytab
> host/<replicafqdn>@<REALM> on the replica? And then kvno
> HTTP/<serverfqdn>@<REALM> ?
> flo
>
> On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via
> FreeIPA-users <freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>> wrote:
>
> That was the /var/log/ipareplica-conncheck.log log file
>
> it does looks like a DNs issue, but im not sure where.
>
> dns resolves the host fine on the host
>
> [root@ipa011 ~]# host ipa011
> ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm>
> has address 10.32.225.7
>
> [root@ipa011 ~]# grep ipa /etc/ipa/default.conf
> host = ipa011.ad.companyx.fm
> <http://ipa011.ad.companyx.fm>
> xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml
> ca_host = ipa010.ad.companyx.fm
> <http://ipa010.ad.companyx.fm>
>
> it's odd as i run the connection check before the
> start of the install, to check ports and routes.
> it works fine.
> replica install works.
> dns install works.
> just the ca installer comes back with this error.
>
> As an additional test i added the dns record for
> this host into IPA before the install. Normally we
> don't need to, but just as a test, but it made no
> difference.
>
>
> We do have new DNS forwarders on the network - these
> are in front of the IPA servers. They are there
> just take the load from the k8s clusters away from
> IPA DNS.
> Would the CA install break if the DNS lookups are
> "proxied" by the DNS forwarders?
> All DNS tests i can think of work via the
> forwarders. The IPA clients (100s) are all fine
> with them.
>
> I will update the client to ignore the forwarders,
> but if you can think of anything else to try?
>
> thanks, Nick
> _______________________________________________
> FreeIPA-users mailing list --
> freeipa-users@lists.fedorahosted.org
> <mailto:freeipa-users@lists.fedorahosted.org>
> To unsubscribe send an email to
> freeipa-users-leave@lists.fedorahosted.org
> <mailto:freeipa-users-leave@lists.fedorahosted.org>
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines:
> https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> Do not reply to spam, report it:
> https://pagure.io/fedora-infrastructure/new_issue
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>