Hi,
I turned on the debugging, but it showed nothing at all when i ran the
conn-check command again. :(
Interestingly this turn up in the http errors log when restarted it (but
before i ran conn-check)
[Thu May 25 09:58:25.521316 2023] [wsgi:error] [pid 2226275:tid 2226275]
ipa: ERROR: Failed to pre-populate LDAP schema cache:
uri=ldapi://%2Frun%2Fslapd-AD-companyx-FM.socket: Unable to retrieve LDAP
schema: Inappropriate authentication: Anonymous access is not allowed.
Is this just a warning? We do turn off full anonymous access for Security
reasons.
Also, when running conn-check, I get the same error if i run the conn-check
command against my new server ipa011 from any other current IPA server
eg. from ipa008 to ipa011, or ipa004 to ipa011.
But not if i run from a current server to another current server. eg.
ipa008 to ipa004 this works fine.
So it seems the issue is just this newly installed IPA server.
#
# from ipa011 to ipa008
#
Check RPC connection to remote master
Connection to
https://ipa008.ad.companyx.fm/ipa/json failed with
Insufficient access: SASL(-1): generic failure: GSSAPI Error: No
credentials were supplied, or the credentials were unavailable or
inaccessible (Credential cache is empty)
Connection to
https://ipa010.ad.companyx.fm/ipa/json failed with
Insufficient access: SASL(-1): generic failure: GSSAPI Error: No
credentials were supplied, or the credentials were unavailable or
inaccessible (Credential cache is empty)
Connection to
https://ipa004dc.ad.companyx.fm/ipa/json failed with
Insufficient access: SASL(-1): generic failure: GSSAPI Error: No
credentials were supplied, or the credentials were unavailable or
inaccessible (Credential cache is empty)
Connection to
https://ipa006dc.ad.companyx.fm/ipa/json failed with
Insufficient access: SASL(-1): generic failure: GSSAPI Error: No
credentials were supplied, or the credentials were unavailable or
inaccessible (Credential cache is empty)
Execute check on remote master
ERROR: Remote master check failed with following error message(s):
invalid 'cn': must be "ipa011.ad.companyx.fm"
timed out waiting for input: auto-logout
#
# from ipa008 to ipa011
#
Check RPC connection to remote master
Connection to
https://ipa011.ad.companyx.fm/ipa/json failed with
Insufficient access: SASL(-1): generic failure: GSSAPI Error: No
credentials were supplied, or the credentials were unavailable or
inaccessible (Credential cache is empty)
Execute check on remote master
ERROR: Remote master check failed with following error message(s):
invalid 'cn': must be "ipa008.ad.companyx.fm"
#
# from ipa008 to ipa010
#
Check RPC connection to remote master
Execute check on remote master
Check connection from master to remote replica 'ipa008.ad.companyx.fm':
Directory Service: Unsecure port (389): OK
Directory Service: Secure port (636): OK
Kerberos KDC: TCP (88): OK
Failed to connect to port 88 udp on 10.32.218.203
Kerberos KDC: UDP (88): WARNING
Kerberos Kpasswd: TCP (464): OK
Failed to connect to port 464 udp on 10.32.218.203
Kerberos Kpasswd: UDP (464): WARNING
HTTP Server: Unsecure port (80): OK
HTTP Server: Secure port (443): OK
The following UDP ports could not be verified as open: 88, 464
This can happen if they are already bound to an application
and ipa-replica-conncheck cannot attach own UDP responder.
Connection from master to replica is OK.
Currently, i am trying to find the python code that spits out the error and
then maybe to tweak it a little to throw some more debug out.
maybe here?
https://github.com/freeipa/freeipa/blob/master/ipaserver/plugins/server.p...
thanks,
Nick
On Thu, 25 May 2023 at 10:13, Florence Blanc-Renaud <flo(a)redhat.com> wrote:
Hi,
On Thu, May 25, 2023 at 9:53 AM Nicholas Cross <nicholas.cross(a)dice.fm>
wrote:
> Hi, thanks for the reply.
>
> This was the process.
> 1. prep
> 2. test connectivity - works ok
> 3. install replica (successfully) - no DNS no CA at this point
> 4. test connectivity - fails.
>
> If i replace step 4 for the following:
>
> 4. install DNS (successfully)
> 5. install CA (fails - with the "invalid 'cn': must be" error)
>
> If we can ignore the step 4 where the comms check fails and concentrate
> on step 5 where the CA fails. I am happy to do so.
> But in step 5. it is the comms check that the `ipa-ca-install` command
> runs that also fails as far as i can see (no expert here myself, though my
> company thinks so LOL)
>
> Successful DNS install logs
http://sprunge.us/7WCtpW
>
> Failed CA install logs
http://sprunge.us/hTVLzL
>
> I believe this error needs to be investigated:
2023-05-25T07:52:07Z INFO Connection to
https://ipa008.ad.companyx.fm/ipa/json failed
with Insufficient access: SASL(-1): generic failure: GSSAPI Error: No credentials were
supplied, or the credentials were unavailable or inaccessible (Credential cache is empty)
We can see that the process is able to connect to the master and get a
cookie but later on GSSAPI auth fails.
Are there any corresponding logs in the master in
/var/log/httpd/error_log? You can enable debug logs on the master with the
following procedure:
- create /etc/ipa/server.conf:
[global]
debug=True
then restart apache with systemctl restart httpd.
You can also run ipa-healthcheck on the master/the replica to find any
config issue.
flo
[root@ipa011 ~]# ipa-ca-install
> Directory Manager (existing master) password:
>
> Run connection check to master
>
> Your system may be partly configured.
> Run /usr/sbin/ipa-server-install --uninstall to clean up.
>
> Connection check failed!
> See /var/log/ipareplica-conncheck.log for more information.
> If the check results are not valid it can be skipped with
> --skip-conncheck parameter.
>
>
>
> thanks,
> Nick
>
> On Thu, 25 May 2023 at 08:18, Florence Blanc-Renaud <flo(a)redhat.com>
> wrote:
>
>> Hi,
>>
>> On Wed, May 24, 2023 at 3:29 PM Nicholas Cross <nicholas.cross(a)dice.fm>
>> wrote:
>>
>>> Hi Flo (and other helpful people on this list),
>>>
>>> After fixing the SID/PAC issue, i am back to looking as to why the
>>> ipa-replica-conncheck fails when installing the CA to a (working) replica.
>>>
>>> I ran your suggested commands and they look fine now.
>>>
>>> [root@ipa011 ~]# kdestroy
>>> kdestroy: No credentials cache found while destroying cache
>>>
>>> [root@ipa011 ~]# kinit -V -kt /etc/krb5.keytab host/
>>> ipa011.ad.companyx.fm(a)AD.companyx.FM
>>> Using default cache: 0
>>> Using principal: host/ipa011.ad.companyx.fm(a)AD.companyx.FM
>>> Using keytab: /etc/krb5.keytab
>>> Authenticated to Kerberos v5
>>>
>>> [root@ipa011 ~]# kvno HTTP/ipa011.ad.companyx.fm(a)AD.companyx.FM
>>> HTTP/ipa011.ad.companyx.fm(a)AD.companyx.FM: kvno = 1
>>>
>>> I can run the following BEFORE i install the replica and it works 100%,
>>> it is part of our pre-reqs in our docs to make sure comms work.
>>>
>>> ipa-replica-conncheck --master=ipa008.ad.companyx.fm
>>> --auto-master-check --realm=AD.companyx.FM --principal=nicholas.cross
>>>
>>> After i install the replica with ipa-replica-install. (no DNS, no CA)
>>> then running the above ipa-replica-conncheck command fails and we see the
>>> errors as described.
>>>
>>
>> ipa-replica-conncheck is intended to be run *before* replica
>> installation, as stated in the man page:
>> ipa-replica-conncheck - Check a replica-master network connection
>> before installation
>>
>> The reason is that it opens server sockets listening on the tested
>> ports, for the master to try to reach the replica. If the replica is
>> already installed, the ports are already in use by the ldap server, the
>> apache web server etc...
>>
>> I understand that you tried to run ipa-replica-conncheck to ensure that
>> ipa-dns-install or ipa-ca-install won't hit any issue. Did you actually try
>> to install the DNS service and the CA service on the replica? If you see
>> errors in those steps we can help troubleshoot, based on the produced logs,
>> but ipa-replica-conncheck won't be of any use here.
>>
>> flo
>>
>>>
>>> Some logs:
>>> 1. before ipa-replica-install - conn check works.
>>>
http://sprunge.us/4yrInD
>>> 2. after successful ipa-replica-install - but conn check fails
>>>
http://sprunge.us/gvqEle
>>>
>>> new replica versions
>>> [root@ipa011 ~]# rpm -qa | grep ipa | sort
>>> almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>>> ipa-client-4.10.1-6.el9.x86_64
>>> ipa-client-common-4.10.1-6.el9.noarch
>>> ipa-common-4.10.1-6.el9.noarch
>>> ipa-healthcheck-0.12-1.el9.noarch
>>> ipa-healthcheck-core-0.12-1.el9.noarch
>>> ipa-selinux-4.9.8-7.el9_0.noarch
>>> ipa-server-4.10.1-6.el9.x86_64
>>> ipa-server-common-4.10.1-6.el9.noarch
>>> ipa-server-dns-4.10.1-6.el9.noarch
>>> libipa_hbac-2.8.2-2.el9.x86_64
>>> python3-ipaclient-4.10.1-6.el9.noarch
>>> python3-ipalib-4.10.1-6.el9.noarch
>>> python3-ipaserver-4.10.1-6.el9.noarch
>>> python3-libipa_hbac-2.8.2-2.el9.x86_64
>>> sssd-ipa-2.8.2-2.el9.x86_64
>>>
>>> current master versions
>>> [nicholas.cross@ipa008 ~]$ rpm -qa | grep ipa | sort
>>> almalinux-logos-ipa-90.5.1-1.1.el9.noarch
>>> ipa-client-4.10.0-8.el9_1.x86_64
>>> ipa-client-common-4.10.0-8.el9_1.noarch
>>> ipa-common-4.10.0-8.el9_1.noarch
>>> ipa-healthcheck-0.9-9.el9.noarch
>>> ipa-healthcheck-core-0.9-9.el9.noarch
>>> ipa-selinux-4.9.8-7.el9_0.noarch
>>> ipa-server-4.10.0-8.el9_1.x86_64
>>> ipa-server-common-4.10.0-8.el9_1.noarch
>>> ipa-server-dns-4.10.0-8.el9_1.noarch
>>> libipa_hbac-2.7.3-4.el9_1.3.x86_64
>>> python3-ipaclient-4.10.0-8.el9_1.noarch
>>> python3-ipalib-4.10.0-8.el9_1.noarch
>>> python3-ipaserver-4.10.0-8.el9_1.noarch
>>> python3-libipa_hbac-2.7.3-4.el9_1.3.x86_64
>>> sssd-ipa-2.7.3-4.el9_1.3.x86_64
>>>
>>> thanks,
>>> Nick
>>>
>>> On Tue, 23 May 2023 at 10:01, Florence Blanc-Renaud <flo(a)redhat.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> the replica-conncheck error means that a call to server_conncheck
>>>> reached the wrong server. ipa-replica-conncheck performs multiple
checks:
>>>> - first from the replica to the existing master (here we seem to be
>>>> good)
>>>> - then from the existing master to the replica, by doing a call to the
>>>> XMLRPC api server_conncheck on the master. If the connection from
replica
>>>> to master fails, another server is tried (in this case, the replica
>>>> launches server_conncheck on itself), but there is a security that
ensures
>>>> the right server is handling the call.
>>>> The logs shows that the connection fails because of SASL auth failure:
>>>> 2023-05-22T18:14:03Z INFO Connection to
>>>>
https://ipa010.ad.companyx.fm/ipa/json failed with Insufficient
>>>> access: SASL(-1): generic failure: GSSAPI Error: No credentials were
>>>> supplied, or the credentials were unavailable or inaccessible
(Credential
>>>> cache is empty)
>>>>
>>>> Are you able to do kinit -kt /etc/krb5.keytab
>>>> host/<replicafqdn>@<REALM> on the replica? And then kvno
>>>> HTTP/<serverfqdn>@<REALM> ?
>>>> flo
>>>>
>>>> On Tue, May 23, 2023 at 9:55 AM Nicholas Cross via FreeIPA-users <
>>>> freeipa-users(a)lists.fedorahosted.org> wrote:
>>>>
>>>>> That was the /var/log/ipareplica-conncheck.log log file
>>>>>
>>>>> it does looks like a DNs issue, but im not sure where.
>>>>>
>>>>> dns resolves the host fine on the host
>>>>>
>>>>> [root@ipa011 ~]# host ipa011
>>>>> ipa011.ad.companyx.fm has address 10.32.225.7
>>>>>
>>>>> [root@ipa011 ~]# grep ipa /etc/ipa/default.conf
>>>>> host = ipa011.ad.companyx.fm
>>>>> xmlrpc_uri =
https://ipa011.ad.companyx.fm/ipa/xml
>>>>> ca_host = ipa010.ad.companyx.fm
>>>>>
>>>>> it's odd as i run the connection check before the start of the
>>>>> install, to check ports and routes. it works fine.
>>>>> replica install works.
>>>>> dns install works.
>>>>> just the ca installer comes back with this error.
>>>>>
>>>>> As an additional test i added the dns record for this host into IPA
>>>>> before the install. Normally we don't need to, but just as a
test, but it
>>>>> made no difference.
>>>>>
>>>>>
>>>>> We do have new DNS forwarders on the network - these are in front of
>>>>> the IPA servers. They are there just take the load from the k8s
clusters
>>>>> away from IPA DNS.
>>>>> Would the CA install break if the DNS lookups are "proxied"
by the
>>>>> DNS forwarders?
>>>>> All DNS tests i can think of work via the forwarders. The IPA
>>>>> clients (100s) are all fine with them.
>>>>>
>>>>> I will update the client to ignore the forwarders, but if you can
>>>>> think of anything else to try?
>>>>>
>>>>> thanks, Nick
>>>>> _______________________________________________
>>>>> FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
>>>>> To unsubscribe send an email to
>>>>> freeipa-users-leave(a)lists.fedorahosted.org
>>>>> Fedora Code of Conduct:
>>>>>
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
>>>>> List Guidelines:
>>>>>
https://fedoraproject.org/wiki/Mailing_list_guidelines
>>>>> List Archives:
>>>>>
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
>>>>> Do not reply to spam, report it:
>>>>>
https://pagure.io/fedora-infrastructure/new_issue
>>>>>
>>>>