So, some good news.
I tried this again today and it works, i turned on the httpd debugging as
suggested on the master and it worked. O_O
The only thing i have changed outside of the work we were doing here was
that i noticed we had a lot of ghost PKI servers yesterday in `pki
securitydomain-show`
So i followed
and removed them.
If this fixed it i don't know, how it could affect it again i dont know,
but it was the only other change we made.
I shall test again on Tuesday to see if we are still ok.
thanks for your continued help on this. have a great weekend.
Nick.
On Fri, 26 May 2023 at 09:35, Florence Blanc-Renaud <flo(a)redhat.com> wrote:
Hi,
in your logs we see the following:
2023-05-25T07:52:07Z DEBUG trying
https://ipa008.ad.companyx.fm/ipa/json
2023-05-25T07:52:07Z
<
https://ipa008.ad.companyx.fm/ipa/json%0D2023-05-25T07:52:07Z> DEBUG New HTTP
connection (ipa008.ad.companyx.fm)
2023-05-25T07:52:07Z DEBUG received Set-Cookie (<class
'list'>)'['ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;path=/ipa;httponly;secure;']'
2023-05-25T07:52:07Z DEBUG storing cookie
'ipa_session=MagBearerToken=YcNKsa8%2bZ%2fVQ8AqCubb5p7Jupgb6nvya2wgS8lYw%2fZKoGctAmsRoh%2boJM7vQNfInsBxx%2fEuf%2fqVlfnKYfWCwwcbU0VH66cM8MqnJhEfKLAgFnNIyG91v%2bmDrsrdCijEWBX8p2PJmqVLIfmvYl2L2ke9FeVPEKnPrq18w1DmZSDDSwV5IOp5JZ71EgTjPoz39Y7mR0LCZ5cXbJ2T93%2b0ZjbcdEsoatFdzkjS383%2f2pLhpRjoTMXCmht%2fXYHjMy47O;'
for principal host/ipa011.ad.companyx.fm(a)AD.companyx.FM
2023-05-25T07:52:07Z INFO Connection to
https://ipa008.ad.companyx.fm/ipa/json failed
with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were
supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
2023-05-25T07:52:07Z DEBUG trying
https://ipa011.ad.companyx.fm/ipa/json
This shows that in a first pass, the replica tries to connect to the
master ipa008 in order to do RPC calls. This corresponds to this piece of
code
https://github.com/freeipa/freeipa/blob/master/install/tools/ipa-replica-...
where the api.Backend.rpcclient.connect() method internally tries to reach
one of the IPA servers (for the details, see
https://github.com/freeipa/freeipa/blob/master/ipalib/rpc.py#L1002, the
available servers are tried sequentially in the *for url in urls:* loop).
In your case, it fails on ipa008 and continues with ipa011 (the replica).
The call to the api server_conncheck is then executed on the local replica
and this explains the error: we don't want to check connection from local
to local, we want to check from ipa008 to ipa011.
The piece of code that needs investigation is why does the connection to
ipa008 fail with GSSAPI error. The debug must be set on the existing
master, in order to see why the authentication fails there.
HTH,
flo
On Thu, May 25, 2023 at 3:56 PM Nicholas Cross <nicholas.cross(a)dice.fm>
wrote:
> Hi Rob,
>
> Thanks for the tip, it came in handy with what i was working on (the
> python code).
>
> Here is the error_log on the new server (ipa011)
>
> [Thu May 25 13:45:38.197358 2023] [mpm_event:notice] [pid 55597:tid
> 55597] AH00492: caught SIGWINCH, shutting down gracefully
> [Thu May 25 13:45:40.008302 2023] [suexec:notice] [pid 55907:tid 55907]
> AH01232: suEXEC mechanism enabled (wrapper: /usr/sbin/suexec)
> [Thu May 25 13:45:40.087736 2023] [lbmethod_heartbeat:notice] [pid
> 55907:tid 55907] AH02282: No slotmem from mod_heartmonitor
> [Thu May 25 13:45:40.094676 2023] [mpm_event:notice] [pid 55907:tid
> 55907] AH00489: Apache/2.4.53 (AlmaLinux) OpenSSL/3.0.7
> mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9 configured -- resuming
> normal operations
> [Thu May 25 13:45:40.094699 2023] [core:notice] [pid 55907:tid 55907]
> AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
> [Thu May 25 13:45:47.384449 2023] [wsgi:error] [pid 55912:tid 56171]
> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb]
> nicholas.cross(a)AD.companyx.FM: ping(): SUCCESS
> [Thu May 25 13:45:47.414961 2023] [wsgi:error] [pid 55914:tid 56174]
> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb]
> nicholas.cross(a)AD.companyx.FM: ping/1(version='2.251'): SUCCESS
> [Thu May 25 13:45:47.445318 2023] [wsgi:error] [pid 55913:tid 56177]
> [remote 10.32.225.7:37118] ipa: INFO: [jsonserver_kerb]
> nicholas.cross(a)AD.companyx.FM: server_conncheck('ipa008.ad.companyx.fm',
> 'ipa011.ad.companyx.fm', version='2.162'): ValidationError
>
> I tweaked the code a little to do some old skool debugging...
> here:
>
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.p...
>
> This enabled me to dump the vars in question to why the error was raised:
>
> ERROR: Remote master check failed with following error message(s):
> invalid 'cn': xx must be "ipa011.ad.companyx.fm" keys:('
> ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm') keys[-2]:
> ipa008.ad.companyx.fm api.env.host:ipa011.ad.companyx.fm
>
> So:
> keys:('ipa008.ad.companyx.fm', 'ipa011.ad.companyx.fm')
> keys[-2]:ipa008.ad.companyx.fm
> api.env.host:ipa011.ad.companyx.fm
>
> As the code states:
>
> if keys[-2] != api.env.host:
> raise errors.ValidationError(
> name='cn', error=_("must be \"%s\"") %
api.env.host)
>
> and we can see that in fact ipa008.ad.companyx.fm !=
> ipa011.ad.companyx.fm
>
> So, that's about as far as i have gotten so far.
>
> Do we think the keys swapped around for some reason?
>
> thanks,
> Nick
>
> On Thu, 25 May 2023 at 13:52, Rob Crittenden <rcritten(a)redhat.com> wrote:
>
>> Florence Blanc-Renaud via FreeIPA-users wrote:
>> > Hi,
>> >
>> > On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross(a)dice.fm
>> > <mailto:nicholas.cross@dice.fm>> wrote:
>> >
>> > Hi, thanks for the reply.
>> >
>> > This was the process.
>> > 1. prep
>> > 2. test connectivity - works ok
>> > 3. install replica (successfully) - no DNS no CA at this point
>> > 4. test connectivity - fails.
>> >
>> > If i replace step 4 for the following:
>> >
>> > 4. install DNS (successfully)
>> > 5. install CA (fails - with the "invalid 'cn': must
be" error)
>> >
>> > If we can ignore the step 4 where the comms check fails and
>> > concentrate on step 5 where the CA fails. I am happy to do so.
>> > But in step 5. it is the comms check that the `ipa-ca-install`
>> > command runs that also fails as far as i can see (no expert here
>> > myself, though my company thinks so LOL)
>> >
>> > Successful DNS install logs
http://sprunge.us/7WCtpW
>> >
>> > Failed CA install logs
http://sprunge.us/hTVLzL
>> >
>> > I believe this error needs to be investigated:
>> >
>> > 2023-05-25T07:52:07Z INFO Connection to
>>
https://ipa008.ad.companyx.fm/ipa/json failed with Insufficient access:
>> SASL(-1): generic failure: GSSAPI Error: No credentials were supplied, or
>> the credentials were unavailable or inaccessible (Credential cache is empty)
>> >
>> > We can see that the process is able to connect to the master and get a
>> > cookie but later on GSSAPI auth fails.
>> >
>> > Are there any corresponding logs in the master in
>> > /var/log/httpd/error_log? You can enable debug logs on the master with
>> > the following procedure:
>> > - create /etc/ipa/server.conf:
>> > [global]
>> > debug=True
>> >
>> > then restart apache with systemctl restart httpd.
>> > You can also run ipa-healthcheck on the master/the replica to find any
>> > config issue.
>>
>> It would also be interesting to see the entries around the
>> 'server_conncheck' call in /var/log/httpd/error_log to see what
>> arguments were passed in (on ipa011) since you're enabling debug mode
>> anyway.
>>
>> rob
>>
>> >
>> > flo
>> >
>> > [root@ipa011 ~]# ipa-ca-install
>> > Directory Manager (existing master) password:
>> >
>> > Run connection check to master
>> >
>> > Your system may be partly configured.
>> > Run /usr/sbin/ipa-server-install --uninstall to clean up.
>> >
>> > Connection check failed!
>> > See /var/log/ipareplica-conncheck.log for more information.
>> > If the check results are not valid it can be skipped with
>> > --skip-conncheck parameter.
>> >
>> >
>> >
>> > thanks,
>> > Nick
>> >
>> > On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <
>> flo(a)redhat.com
>> > <mailto:flo@redhat.com>> wrote:
>> >
>> > Hi,
>> >
>> > On Wed, May 24, 2023 at 3:29 PM Nicholas Cross
>> > <nicholas.cross(a)dice.fm
<mailto:nicholas.cross@dice.fm>>
>> wrote:
>> >
>> > Hi Flo (and other helpful people on this list),
>> >
>> > After fixing the SID/PAC issue, i am back to looking as to
>> > why the ipa-replica-conncheck fails when installing the CA
>> > to a (working) replica.
>> >
>> > I ran your suggested commands and they look fine now.
>> >
>> > [root@ipa011 ~]# kdestroy
>> > kdestroy: No credentials cache found while destroying cache
>> >
>> > [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab
>> > host/ipa011.ad.companyx.fm(a)AD.companyx.FM
>> > <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>> > Using default cache: 0
>> > Using principal: host/ipa011.ad.companyx.fm(a)AD.companyx.FM
>> > <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>> > Using keytab: /etc/krb5.keytab
>> > Authenticated to Kerberos v5
>> >
>> > [root@ipa011 ~]# kvno
>> > HTTP/ipa011.ad.companyx.fm(a)AD.companyx.FM
>> > <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>
>> > HTTP/ipa011.ad.companyx.fm(a)AD.companyx.FM
>> > <mailto:ipa011.ad.companyx.fm@AD.companyx.FM>: kvno = 1
>> >
>> > I can run the following BEFORE i install the replica and it
>> > works 100%, it is part of our pre-reqs in our docs to make
>> > sure comms work.
>> >
>> > ipa-replica-conncheck --master=ipa008.ad.companyx.fm
>> > <
http://ipa008.ad.companyx.fm> --auto-master-check
>> > --realm=AD.companyx.FM <
http://AD.companyx.FM>
>> > --principal=nicholas.cross
>> >
>> > After i install the replica with ipa-replica-install. (no
>> > DNS, no CA) then running the above ipa-replica-conncheck
>> > command fails and we see the errors as described.
>> >
>> >
>> > ipa-replica-conncheck is intended to be run *before* replica
>> > installation, as stated in the man page:
>> > ipa-replica-conncheck - Check a replica-master network
>> > connection before installation
>> >
>> > The reason is that it opens server sockets listening on the
>> > tested ports, for the master to try to reach the replica. If
>> the
>> > replica is already installed, the ports are already in use by
>> > the ldap server, the apache web server etc...
>> >
>> > I understand that you tried to run ipa-replica-conncheck to
>> > ensure that ipa-dns-install or ipa-ca-install won't hit any
>> > issue. Did you actually try to install the DNS service and the
>> > CA service on the replica? If you see errors in those steps we
>> > can help troubleshoot, based on the produced logs, but
>> > ipa-replica-conncheck won't be of any use here.
>> >
>> > flo
>> >
>> >
>> > Some logs:
>> > 1. before ipa-replica-install - conn check works.
>> >
http://sprunge.us/4yrInD
>> > 2. after successful ipa-replica-install - but conn check
>> fails
>> >
http://sprunge.us/gvqEle
>> >
>> > new replica versions
>> > [root@ipa011 ~]# rpm -qa | grep ipa | sort
>> > almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>> > ipa-client-4.10.1-6.el9.x86_64
>> > ipa-client-common-4.10.1-6.el9.noarch
>> > ipa-common-4.10.1-6.el9.noarch
>> > ipa-healthcheck-0.12-1.el9.noarch
>> > ipa-healthcheck-core-0.12-1.el9.noarch
>> > ipa-selinux-4.9.8-7.el9_0.noarch
>> > ipa-server-4.10.1-6.el9.x86_64
>> > ipa-server-common-4.10.1-6.el9.noarch
>> > ipa-server-dns-4.10.1-6.el9.noarch
>> > libipa_hbac-2.8.2-2.el9.x86_64
>> > python3-ipaclient-4.10.1-6.el9.noarch
>> > python3-ipalib-4.10.1-6.el9.noarch
>> > python3-ipaserver-4.10.1-6.el9.noarch
>> > python3-libipa_hbac-2.8.2-2.el9.x86_64
>> > sssd-ipa-2.8.2-2.el9.x86_64
>> >
>> > current master versions
>> > [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort
>> > almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>> > ipa-client-4.10.0-8.el9_1.x86_64
>> > ipa-client-common-4.10.0-8.el9_1.noarch
>> > ipa-common-4.10.0-8.el9_1.noarch
>> > ipa-healthcheck-0.9-9.el9.noarch
>> > ipa-healthcheck-core-0.9-9.el9.noarch
>> > ipa-selinux-4.9.8-7.el9_0.noarch
>> > ipa-server-4.10.0-8.el9_1.x86_64
>> > ipa-server-common-4.10.0-8.el9_1.noarch
>> > ipa-server-dns-4.10.0-8.el9_1.noarch
>> > libipa_hbac-2.7.3-4.el9_1.3.x86_64
>> > python3-ipaclient-4.10.0-8.el9_1.noarch
>> > python3-ipalib-4.10.0-8.el9_1.noarch
>> > python3-ipaserver-4.10.0-8.el9_1.noarch
>> > python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
>> > sssd-ipa-2.7.3-4.el9_1.3.x86_64
>> >
>> > thanks,
>> > Nick
>> >
>> > On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud
>> > <flo(a)redhat.com <mailto:flo@redhat.com>> wrote:
>> >
>> > Hi,
>> >
>> > the replica-conncheck error means that a call to
>> > server_conncheck reached the wrong server.
>> > ipa-replica-conncheck performs multiple checks:
>> > - first from the replica to the existing master (here
>> we
>> > seem to be good)
>> > - then from the existing master to the replica, by
>> doing
>> > a call to the XMLRPC api server_conncheck on the
>> master.
>> > If the connection from replica to master fails, another
>> > server is tried (in this case, the replica launches
>> > server_conncheck on itself), but there is a security
>> > that ensures the right server is handling the call.
>> > The logs shows that the connection fails because of
>> SASL
>> > auth failure:
>> > 2023-05-22T18:14:03Z INFO Connection to
>> >
https://ipa010.ad.companyx.fm/ipa/json failed with
>> > Insufficient access: SASL(-1): generic failure: GSSAPI
>> > Error: No credentials were supplied, or the credentials
>> > were unavailable or inaccessible (Credential cache is
>> empty)
>> >
>> > Are you able to do kinit -kt /etc/krb5.keytab
>> > host/<replicafqdn>@<REALM> on the replica? And
then
>> kvno
>> > HTTP/<serverfqdn>@<REALM> ?
>> > flo
>> >
>> > On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via
>> > FreeIPA-users <freeipa-users(a)lists.fedorahosted.org
>> > <mailto:freeipa-users@lists.fedorahosted.org>>
wrote:
>> >
>> > That was the /var/log/ipareplica-conncheck.log log
>> file
>> >
>> > it does looks like a DNs issue, but im not sure
>> where.
>> >
>> > dns resolves the host fine on the host
>> >
>> > [root@ipa011 ~]# host ipa011
>> > ipa011.ad.companyx.fm <
>>
http://ipa011.ad.companyx.fm>
>> > has address 10.32.225.7
>> >
>> > [root@ipa011 ~]# grep ipa /etc/ipa/default.conf
>> > host = ipa011.ad.companyx.fm
>> > <
http://ipa011.ad.companyx.fm>
>> > xmlrpc_uri =
https://ipa011.ad.companyx.fm/ipa/xml
>> > ca_host = ipa010.ad.companyx.fm
>> > <
http://ipa010.ad.companyx.fm>
>> >
>> > it's odd as i run the connection check before the
>> > start of the install, to check ports and routes.
>> > it works fine.
>> > replica install works.
>> > dns install works.
>> > just the ca installer comes back with this error.
>> >
>> > As an additional test i added the dns record for
>> > this host into IPA before the install. Normally we
>> > don't need to, but just as a test, but it made no
>> > difference.
>> >
>> >
>> > We do have new DNS forwarders on the network -
>> these
>> > are in front of the IPA servers. They are there
>> > just take the load from the k8s clusters away from
>> > IPA DNS.
>> > Would the CA install break if the DNS lookups are
>> > "proxied" by the DNS forwarders?
>> > All DNS tests i can think of work via the
>> > forwarders. The IPA clients (100s) are all fine
>> > with them.
>> >
>> > I will update the client to ignore the forwarders,
>> > but if you can think of anything else to try?
>> >
>> > thanks, Nick
>> > _______________________________________________
>> > FreeIPA-users mailing list --
>> > freeipa-users(a)lists.fedorahosted.org
>> > <mailto:freeipa-users@lists.fedorahosted.org>
>> > To unsubscribe send an email to
>> > freeipa-users-leave(a)lists.fedorahosted.org
>> > <mailto:freeipa-users-leave@lists.fedorahosted.org
>> >
>> > Fedora Code of Conduct:
>> >
>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines:
>> >
>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
>> >
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>> > Do not reply to spam, report it:
>> >
https://pagure.io/fedora-infrastructure/new_issue
>> >
>> >
>> > _______________________________________________
>> > FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>> > To unsubscribe send an email to
>> freeipa-users-leave(a)lists.fedorahosted.org
>> > Fedora Code of Conduct:
>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>> > List Guidelines:
>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>> > List Archives:
>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>> > Do not reply to spam, report it:
>>
https://pagure.io/fedora-infrastructure/new_issue
>> >
>>
>>