So, some good news.

I tried this again today and it works,  i turned on the httpd debugging as suggested on the master and it worked. O_O

The only thing i have changed outside of the work we were doing here was that i noticed we had a lot of ghost PKI servers yesterday in `pki securitydomain-show`
So i followed https://access.redhat.com/solutions/6615411 and removed them.

If this fixed it i don't know, how it could affect it again i dont know, but it was the only other change we made.

I shall test again on Tuesday to see if we are still ok.

thanks for your continued help on this.  have a great weekend.
Nick.


On Fri, 26 May 2023 at 09:35, Florence Blanc-Renaud <flo@redhat.com> wrote:
Hi,

in your logs we see the following:
2023-05-25T07:52:07Z DEBUG trying https://ipa008.ad.companyx.fm/ipa/json
2023-05-25T07:52:07Z DEBUG New HTTP connection (ipa008.ad.companyx.fm)
2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class 'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']'
2023-05-25T07:52:07Z DEBUG storing cookie 'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;' for principal host/ipa011.ad.companyx.fm@AD.companyx.FM
2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
2023-05-25T07:52:07Z DEBUG trying https://ipa011.ad.companyx.fm/ipa/json

This shows that in a first pass, the replica tries to connect to the master ipa008 in order to do RPC calls. This corresponds to this piece of code https://github.com/freeipa/freeipa/blob/master/install/tools/ipa-replica-conncheck.in#L560-L567 where the api.Backend.rpcclient.connect() method internally tries to reach one of the IPA servers (for the details, see https://github.com/freeipa/freeipa/blob/master/ipalib/rpc.py#L1002, the available servers are tried sequentially in the for url in urls: loop). In your case, it fails on ipa008 and continues with ipa011 (the replica). The call to the api server_conncheck is then executed on the local replica and this explains the error: we don't want to check connection from local to local, we want to check from ipa008 to ipa011.

The piece of code that needs investigation is why does the connection to ipa008 fail with GSSAPI error. The debug must be set on the existing master, in order to see why the authentication fails there.

HTH,
flo

On Thu, May 25, 2023 at 3:56 PM Nicholas Cross <nicholas.cross@dice.fm> wrote:
Hi Rob,

Thanks for the tip, it came in handy with what i was working on (the python code).

Here is the error_log on the new server (ipa011)

[Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid 55597] AH00492: caught SIGWINCH, shutting down gracefully
[Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907] AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
[Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor
[Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming normal operations
[Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
[Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping(): SUCCESS
[Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: ping/1(version='2.251'): SUCCESS
[Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177] [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb] nicholas.cross@AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm', version='2.162'): ValidationError

I tweaked the code a little to do some old skool debugging...

This enabled me to dump the vars in question to why the error was raised:

ERROR: Remote master check failed with following error message(s):
invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm

So:
keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') 
As the code states:

if keys[-2] != api.env.host:
            raise errors.ValidationError(
                name='cn', error=_("must be \"%s\"") % api.env.host)

and we can see that in fact ipa008.ad.companyx.fm != ipa011.ad.companyx.fm

So, that's about as far as i have gotten so far.

Do we think the keys swapped around for some reason?

thanks,
Nick

On Thu, 25 May 2023 at 13:52, Rob Crittenden <rcritten@redhat.com> wrote:
Florence Blanc-Renaud via FreeIPA-users wrote:
> Hi,
>
> On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross@dice.fm
> <mailto:nicholas.cross@dice.fm>> wrote:
>
>     Hi, thanks for the reply.
>
>     This was the process.
>     1. prep
>     2. test connectivity - works ok
>     3. install replica (successfully) - no DNS no CA at this point
>     4. test connectivity - fails.
>
>     If i replace step 4 for the following:
>
>     4. install DNS (successfully)
>     5. install CA (fails - with the "invalid 'cn': must be" error)
>
>     If we can ignore the step 4 where the comms check fails and
>     concentrate on step 5 where the CA fails. I am happy to do so.
>     But in step 5. it is the comms check that the `ipa-ca-install`
>     command runs that also fails as far as i can see (no expert here
>     myself, though my company thinks so LOL)
>
>     Successful DNS install logs http://sprunge.us/7WCtpW 
>
>     Failed CA install logs  http://sprunge.us/hTVLzL
>
> I believe this error needs to be investigated:
>
> 2023-05-25T07:52:07Z INFO Connection to https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
>
> We can see that the process is able to connect to the master and get a
> cookie but later on GSSAPI auth fails.
>
> Are there any corresponding logs in the master in
> /var/log/httpd/error_log? You can enable debug logs on the master with
> the following procedure:
> - create /etc/ipa/server.conf:
> [global]
> debug=True
>
> then restart apache with systemctl restart httpd.
> You can also run ipa-healthcheck on the master/the replica to find any
> config issue.

It would also be interesting to see the entries around the
'server_conncheck' call in /var/log/httpd/error_log to see what
arguments were passed in (on ipa011) since you're enabling debug mode
anyway.

rob

>
> flo
>
>     [root@ipa011 ~]# ipa-ca-install
>     Directory Manager (existing master) password:
>
>     Run connection check to master
>
>     Your system may be partly configured.
>     Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
>     Connection check failed!
>     See /var/log/ipareplica-conncheck.log for more information.
>     If the check results are not valid it can be skipped with
>     --skip-conncheck parameter.
>
>
>
>     thanks,
>     Nick
>
>     On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo@redhat.com
>     <mailto:flo@redhat.com>> wrote:
>
>         Hi,
>
>         On Wed, May 24, 2023 at 3:29 PM Nicholas Cross
>         <nicholas.cross@dice.fm <mailto:nicholas.cross@dice.fm>> wrote:
>
>             Hi Flo (and other helpful people on this list),
>
>             After fixing the SID/PAC issue, i am back to looking as to
>             why the ipa-replica-conncheck fails when installing the CA
>             to a (working) replica.
>
>             I ran your suggested commands and they look fine now.
>
>             [root@ipa011 ~]# kdestroy
>             kdestroy: No credentials cache found while destroying cache
>
>             [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab
>             host/ipa011.ad.companyx.fm@AD.companyx.FM
>             <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>             Using default cache: 0
>             Using principal: host/ipa011.ad.companyx.fm@AD.companyx.FM
>             <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>             Using keytab: /etc/krb5.keytab
>             Authenticated to Kerberos v5
>
>             [root@ipa011 ~]# kvno
>             HTTP/ipa011.ad.companyx.fm@AD.companyx.FM
>             <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>             HTTP/ipa011.ad.companyx.fm@AD.companyx.FM
>             <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1
>
>             I can run the following BEFORE i install the replica and it
>             works 100%, it is part of our pre-reqs in our docs to make
>             sure comms work.
>
>             ipa-replica-conncheck --master=ipa008.ad.companyx.fm
>             <http://ipa008.ad.companyx.fm> --auto-master-check
>             --realm=AD.companyx.FM <http://AD.companyx.FM>
>             --principal=nicholas.cross
>
>             After i install the replica with ipa-replica-install. (no
>             DNS, no CA) then running the above ipa-replica-conncheck
>             command fails and we see the errors as described.
>
>
>         ipa-replica-conncheck is intended to be run *before* replica
>         installation, as stated in the man page:
>                ipa-replica-conncheck - Check a replica-master network
>         connection before installation
>
>         The reason is that it opens server sockets listening on the
>         tested ports, for the master to try to reach the replica. If the
>         replica is already installed, the ports are already in use by
>         the ldap server, the apache web server etc...
>
>         I understand that you tried to run ipa-replica-conncheck to
>         ensure that ipa-dns-install or ipa-ca-install won't hit any
>         issue. Did you actually try to install the DNS service and the
>         CA service on the replica? If you see errors in those steps we
>         can help troubleshoot, based on the produced logs, but
>         ipa-replica-conncheck won't be of any use here.
>
>         flo
>
>
>             Some logs:
>             1. before ipa-replica-install - conn check works.
>             http://sprunge.us/4yrInD
>             2. after successful ipa-replica-install - but conn check fails
>             http://sprunge.us/gvqEle
>
>             new replica versions
>             [root@ipa011 ~]# rpm -qa | grep ipa | sort
>             almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>             ipa-client-4.10.1-6.el9.x86_64
>             ipa-client-common-4.10.1-6.el9.noarch
>             ipa-common-4.10.1-6.el9.noarch
>             ipa-healthcheck-0.12-1.el9.noarch
>             ipa-healthcheck-core-0.12-1.el9.noarch
>             ipa-selinux-4.9.8-7.el9_0.noarch
>             ipa-server-4.10.1-6.el9.x86_64
>             ipa-server-common-4.10.1-6.el9.noarch
>             ipa-server-dns-4.10.1-6.el9.noarch
>             libipa_hbac-2.8.2-2.el9.x86_64
>             python3-ipaclient-4.10.1-6.el9.noarch
>             python3-ipalib-4.10.1-6.el9.noarch
>             python3-ipaserver-4.10.1-6.el9.noarch
>             python3-libipa_hbac-2.8.2-2.el9.x86_64
>             sssd-ipa-2.8.2-2.el9.x86_64
>
>             current master versions
>             [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort
>             almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>             ipa-client-4.10.0-8.el9_1.x86_64
>             ipa-client-common-4.10.0-8.el9_1.noarch
>             ipa-common-4.10.0-8.el9_1.noarch
>             ipa-healthcheck-0.9-9.el9.noarch
>             ipa-healthcheck-core-0.9-9.el9.noarch
>             ipa-selinux-4.9.8-7.el9_0.noarch
>             ipa-server-4.10.0-8.el9_1.x86_64
>             ipa-server-common-4.10.0-8.el9_1.noarch
>             ipa-server-dns-4.10.0-8.el9_1.noarch
>             libipa_hbac-2.7.3-4.el9_1.3.x86_64
>             python3-ipaclient-4.10.0-8.el9_1.noarch
>             python3-ipalib-4.10.0-8.el9_1.noarch
>             python3-ipaserver-4.10.0-8.el9_1.noarch
>             python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
>             sssd-ipa-2.7.3-4.el9_1.3.x86_64
>
>             thanks,
>             Nick
>
>             On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud
>             <flo@redhat.com <mailto:flo@redhat.com>> wrote:
>
>                 Hi,
>
>                 the replica-conncheck error means that a call to
>                 server_conncheck reached the wrong server.
>                 ipa-replica-conncheck performs multiple checks:
>                 - first from the replica to the existing master (here we
>                 seem to be good)
>                 - then from the existing master to the replica, by doing
>                 a call to the XMLRPC api server_conncheck on the master.
>                 If the connection from replica to master fails, another
>                 server is tried (in this case, the replica launches
>                 server_conncheck on itself), but there is a security
>                 that ensures the right server is handling the call.
>                 The logs shows that the connection fails because of SASL
>                 auth failure:
>                 2023-05-22T18:14:03Z INFO Connection to
>                 https://ipa010.ad.companyx.fm/ipa/json failed with
>                 Insufficient access: SASL(-1): generic failure: GSSAPI
>                 Error: No credentials were supplied, or the credentials
>                 were unavailable or inaccessible (Credential cache is empty)
>
>                 Are you able to do kinit -kt /etc/krb5.keytab
>                 host/<replicafqdn>@<REALM> on the replica? And then kvno
>                 HTTP/<serverfqdn>@<REALM> ?
>                 flo
>
>                 On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via
>                 FreeIPA-users <freeipa-users@lists.fedorahosted.org
>                 <mailto:freeipa-users@lists.fedorahosted.org>> wrote:
>
>                     That was the /var/log/ipareplica-conncheck.log log file
>
>                     it does looks like a DNs issue, but im not sure where.
>
>                     dns resolves the host fine on the host
>
>                     [root@ipa011 ~]# host ipa011
>                     ipa011.ad.companyx.fm <http://ipa011.ad.companyx.fm>
>                     has address 10.32.225.7
>
>                     [root@ipa011 ~]# grep ipa /etc/ipa/default.conf
>                     host = ipa011.ad.companyx.fm
>                     <http://ipa011.ad.companyx.fm>
>                     xmlrpc_uri = https://ipa011.ad.companyx.fm/ipa/xml
>                     ca_host = ipa010.ad.companyx.fm
>                     <http://ipa010.ad.companyx.fm>
>
>                     it's odd as i run the connection check before the
>                     start of the install, to check ports and routes. 
>                      it works fine.
>                     replica install works.
>                     dns install works.
>                     just the ca installer comes back with this error.
>
>                     As an additional test i added the dns record for
>                     this host into IPA before the install.  Normally we
>                     don't need to, but just as a test, but it made no
>                     difference.
>
>
>                     We do have new DNS forwarders on the network - these
>                     are in front of the IPA servers.   They are there
>                     just take the load from the k8s clusters away from
>                     IPA DNS.
>                     Would the CA install break if the DNS lookups are
>                     "proxied" by the DNS forwarders?
>                     All DNS tests i can think of work via the
>                     forwarders.  The IPA clients (100s) are all fine
>                     with them.
>
>                     I will update the client to ignore the forwarders,
>                     but if you can think of anything else to try?
>
>                     thanks, Nick
>                     _______________________________________________
>                     FreeIPA-users mailing list --
>                     freeipa-users@lists.fedorahosted.org
>                     <mailto:freeipa-users@lists.fedorahosted.org>
>                     To unsubscribe send an email to
>                     freeipa-users-leave@lists.fedorahosted.org
>                     <mailto:freeipa-users-leave@lists.fedorahosted.org>
>                     Fedora Code of Conduct:
>                     https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>                     List Guidelines:
>                     https://fedoraproject.org/wiki/Mailing_list_guidelines
>                     List Archives:
>                     https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
>                     Do not reply to spam, report it:
>                     https://pagure.io/fedora-infrastructure/new_issue
>
>
> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
>