I have an inherited IPA domain that is a subdomain of an active directory domain, e.g.
ipa.ad1.com as a child of
ad1.com. The IPA domain has AD Trust enabled and a one way
domain trust to another AD sub domain, e.g. we want to use user logins from the AD domain
users.ad2.com which is a child domain of
ad2.com. We are also using AD security group
from the
user.ad2.com domain to apply group based access control. e.g. we are using
simple authentication on SSSD to limit who can login and using AD groups to define sudo
access. This users domain and AD servers is managed by another team.
Everything was working for some time and then we started seeing intermittent problems with
authentication, a quick restart of the IPA server would resolve the problem temporarily,
but then it would stop again. Even if we could login using SSH keys the sudo access would
not work, it would appear to lose group membership details.
I have recently updated all of the IPA nodes to RHEL9 and made sure that DNS is updated
correctly.
The sssd.conf configuration on the IPA server looks as follows
[
domain/ipa.ad1.com]
debug_level = 6
id_provider = ipa
ipa_server =
ipa-3.ipa.ad1.com
ipa_domain =
ipa.ad1.com
ipa_hostname =
ipa-3.ipa.ad1.com
auth_provider = ipa
chpass_provider = ipa
access_provider = ipa
cache_credentials = True
ldap_tls_cacert = /etc/ipa/ca.crt
krb5_store_password_if_offline = True
sudo_provider = ipa
autofs_provider = ipa
subdomains_provider = ipa
session_provider = ipa
hostid_provider = ipa
ipa_server_mode = True
subdomain_homedir = /home/%u
default_shell = /bin/bash
override_shell = /bin/bash
[sssd]
services = nss, pam, sudo, ifp
domains =
ipa.ad1.com
domain_resolution_order =
users.ad2.com
[nss]
homedir_substring = /home
[pam]
[sudo]
[autofs]
[ssh]
[pac]
[ifp]
allowed_uids = ipaapi, root
[session_recording]
I have debug level 6 enabled on SSSD and when I check the domain status I see the
following more often than not. The
ad2.com forest domains are offline. They go online
and then as soon as someone tries to login again then either both or just the
users.ad2.com domain go offline which causes the login to fail.
ipa.ad1.com Online status: Online
ad1.com Online status: Online
ad2.com Online status: Offline
users.ad2.com Online status: Offline
When I look at the SSSD domain logs I see the following (I have replaced internal domain
names or hostname)
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_senders_lookup] (0x2000): Looking for
identity of sender [sssd.ifp]
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain
ipa.ad1.com is Active
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain
ad1.com is Active
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sss_domain_get_state] (0x1000): Domain
AD2.COM is Active
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400):
sssd.DataProvider.Backend.IsOnline: Success
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_dispatch] (0x4000): Dispatching.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162]
EOF received, client finished
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162]
Child responded: 0 [
FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162]
expire timeout is 900
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x1000): [RID#1162]
the connection will expire at 1686152455
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162]
Executing sasl bind mech: GSSAPI, user: AUTH$
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162]
ldap_sasl_interactive_bind_s failed (-2)[Local error]
********************** BACKTRACE DUMP ENDS HERE *********************************
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended
failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor
code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child
[9519] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162]
Unable to establish connection [1432158227]: Authentication Failed
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162]
Extended failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS
failure. Minor code may provide more information (Server not found in Kerberos
database)]
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x1000): [RID#1162]
Waiting for child [9519].
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162]
child [9519] finished successfully.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040):
[RID#1162] Unable to establish connection [1432158227]: Authentication Failed
********************** BACKTRACE DUMP ENDS HERE *********************************
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 3268 of server 'ad1-dc-1.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 3268 of duplicate server 'ad1-dc-1.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The
status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100):
[RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'ad1-dc-3.ad1.com' as 'resolving name'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100):
[RID#1162] Trying to resolve AAAA record of 'ad1-dc-3.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162]
No more address families to retry
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100):
[RID#1162] Trying to resolve A record of 'ad1-dc-3.ad1.com' in DNS
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162]
Deleting request watch
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'ad1-dc-3.ad1.com' as 'name resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162]
Found address for server
ad1-dc-3.ad1.com: [172.28.8.7] TTL 3600
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162]
Constructed uri 'ldap://ad1-dc-3.ad1.com'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162]
Constructed GC uri 'ldap://ad1-dc-3.ad1.com:3268'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400):
[RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162]
calling ldap_search_ext with [(objectclass=*)][].
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400):
[RID#1162] Search result: Success(0), no errmsg set
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100):
[RID#1162] Setting AD compatibility level to [7]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100):
[RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting
kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$,
AD2.COM, 86400)
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The
status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162]
Found address for server
dcc01.ad2.com: [10.194.34.10] TTL 2107
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162]
buffer size: 73
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162]
Setting 8 seconds timeout for TGT child
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All
data has been sent!
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF
received, client finished
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child
responded: 0 [
FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire
timeout is 900
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing
sasl bind mech: GSSAPI, user: AUTH$
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162]
ldap_sasl_interactive_bind_s failed (-2)[Local error]
* ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended
failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor
code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child
[9522] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162]
Unable to establish connection [1432158227]: Authentication Failed
* ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 3268 of server 'ad1-dc-3.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 3268 of duplicate server 'ad1-dc-3.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The
status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100):
[RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'ad1-dc-2.ad1.com' as 'resolving name'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_files_send] (0x0100):
[RID#1162] Trying to resolve AAAA record of 'ad1-dc-2.ad1.com' in files
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_next] (0x0200): [RID#1162]
No more address families to retry
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolv_gethostbyname_dns_query] (0x0100):
[RID#1162] Trying to resolve A record of 'ad1-dc-2.ad1.com' in DNS
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [request_watch_destructor] (0x0400): [RID#1162]
Deleting request watch
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'ad1-dc-2.ad1.com' as 'name resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162]
Found address for server
ad1-dc-2.ad1.com: [172.28.8.6] TTL 3600
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162]
Constructed uri 'ldap://ad1-dc-2.ad1.com'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ad_resolve_callback] (0x0100): [RID#1162]
Constructed GC uri 'ldap://ad1-dc-2.ad1.com:3268'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sssd_async_socket_init_send] (0x0400):
[RID#1162] Setting 6 seconds timeout [ldap_network_timeout] for connecting
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_ext_step] (0x0400): [RID#1162]
calling ldap_search_ext with [(objectclass=*)][].
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_generic_op_finished] (0x0400):
[RID#1162] Search result: Success(0), no errmsg set
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100):
[RID#1162] Setting AD compatibility level to [7]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_server_opts_from_rootdse] (0x0100):
[RID#1162] Will look for schema at [CN=Schema,CN=Configuration,DC=ad1,DC=com]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_kinit_send] (0x0400): [RID#1162] Attempting
kinit (/var/lib/sss/keytabs/AD2.COM.keytab, AUTH$,
AD2.COM, 86400)
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [resolve_srv_send] (0x0200): [RID#1162] The
status of SRV lookup is resolved
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_process] (0x0200): [RID#1162]
Found address for server
dcc01.ad2.com: [10.194.34.10] TTL 2107
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [create_tgt_req_send_buffer] (0x0400): [RID#1162]
buffer size: 73
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_tgt_child_timeout] (0x0400): [RID#1162]
Setting 8 seconds timeout for TGT child
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_write_pipe_handler] (0x0400): [RID#1162] All
data has been sent!
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [_read_pipe_handler] (0x0400): [RID#1162] EOF
received, client finished
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_get_tgt_recv] (0x0400): [RID#1162] Child
responded: 0 [
FILE:/var/lib/sss/db/ccache_AD2.COM], expired on [1686187555]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_auth_step] (0x0100): [RID#1162] expire
timeout is 900
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0100): [RID#1162] Executing
sasl bind mech: GSSAPI, user: AUTH$
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0020): [RID#1162]
ldap_sasl_interactive_bind_s failed (-2)[Local error]
* ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sasl_bind_send] (0x0080): [RID#1162] Extended
failure message: [SASL(-1): generic failure: GSSAPI Error: Unspecified GSS failure. Minor
code may provide more information (Server not found in Kerberos database)]
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child
[9523] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_cli_connect_recv] (0x0040): [RID#1162]
Unable to establish connection [1432158227]: Authentication Failed
* ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No
available servers for service 'sd_gc_AD2.COM'
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [_be_fo_set_port_status] (0x8000):
[RID#1162] Setting status: PORT_NOT_WORKING. Called from:
src/providers/ldap/sdap_async_connection.c: sdap_cli_connect_recv: 2136
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162]
Marking port 3268 of server 'ad1-dc-2.ad1.com' as 'not working'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162]
Marking port 3268 of duplicate server 'ad1-dc-2.ad1.com' as 'not working'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_handle_release] (0x2000): [RID#1162]
Trace: sh[0x556c865496d0], connected[1], ops[(nil)], ldap[0x556c865f89b0],
destructor_lock[0], release_memory[0]
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [remove_connection_callback] (0x4000):
[RID#1162] Successfully removed connection callback.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000):
[RID#1162] attempting failover retry on op #1
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_step] (0x4000):
[RID#1162] beginning to connect
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100):
[RID#1162] Trying to resolve service 'sd_gc_AD2.COM'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162]
Status of server 'ad1-dc-1.ad1.com' is 'name resolved'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port
status of port 3268 for server 'ad1-dc-1.ad1.com' is 'not working'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD
is unable to complete the full connection request, this internal status does not
necessarily indicate network port issues.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162]
Status of server 'ad1-dc-3.ad1.com' is 'name resolved'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port
status of port 3268 for server 'ad1-dc-3.ad1.com' is 'not working'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD
is unable to complete the full connection request, this internal status does not
necessarily indicate network port issues.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_server_status] (0x1000): [RID#1162]
Status of server 'ad1-dc-2.ad1.com' is 'name resolved'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x1000): [RID#1162] Port
status of port 3268 for server 'ad1-dc-2.ad1.com' is 'not working'
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD
is unable to complete the full connection request, this internal status does not
necessarily indicate network port issues.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020):
[RID#1162] No available servers for service 'sd_gc_AD2.COM'
********************** BACKTRACE DUMP ENDS HERE *********************************
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162]
Failed to connect to server, but ignore mark offline is enabled.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0080):
[RID#1162] Subdomain lookup failed, will try to reset subdomain.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trusted_dom_setup_1way] (0x0400):
[RID#1162] Will re-fetch keytab for
USERS.AD2.COM
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_getkeytab_send] (0x0400): [RID#1162]
Retrieving keytab for AUTH$(a)AD2.COM from
ipa-3.ipa.ad1.com into
/var/lib/sss/keytabs/AD2.COM.keytabIw4R58 using ccache
/var/lib/sss/db/ccache_AUTH.SSDIS.LOC
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sbus_issue_request_done] (0x0400):
sssd.DataProvider.Backend.IsOnline: Success
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [child_sig_handler] (0x0100): [RID#1162] child
[9524] finished successfully.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400):
[RID#1162] Keytab successfully retrieved to /var/lib/sss/keytabs/AD2.COM.keytabIw4R58
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400):
[RID#1162] Keytab /var/lib/sss/keytabs/AD2.COM.keytabIw4R58 contains the expected
principals
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_server_trust_1way_kt_done] (0x0400):
[RID#1162] Established trust context for
USERS.AD2.COM
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_retried] (0x0400): [RID#1162]
Subdomain re-set, will retry lookup
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'dcf01.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'dcf01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'dcf01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'dcs02.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'dcs02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'dcs02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'dcf02.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'dcf02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'dcf02.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'dcc01.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'dcc01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'dcc01.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'DCC02.USERS.AD2.COM' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'DCC02.USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_server_common_status] (0x0100): [RID#1162]
Marking server 'dcs03.users.ad2.com' as 'name not resolved'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 389 of server 'dcs03.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0400): [RID#1162] Marking
port 389 of duplicate server 'dcs03.users.ad2.com' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [set_srv_data_status] (0x0100): [RID#1162]
Marking SRV lookup of service 'sd_gc_USERS.AD2.COM' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_set_port_status] (0x0100): [RID#1162] Marking
port 0 of server '(no name)' as 'neutral'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_step] (0x0400):
[RID#1162] Looking up AD account
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0100): [RID#1162]
Trying to resolve service 'sd_gc_AD2.COM'
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [get_port_status] (0x0080): [RID#1162] SSSD is
unable to complete the full connection request, this internal status does not necessarily
indicate network port issues.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [fo_resolve_service_send] (0x0020): [RID#1162] No
available servers for service 'sd_gc_AD2.COM'
* ... skipping repetitive backtrace ...
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400): [RID#1162]
Failed to connect to server, but ignore mark offline is enabled.
(2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040):
[RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
********************** PREVIOUS MESSAGE WAS TRIGGERED BY THE FOLLOWING BACKTRACE:
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_resolve_server_done] (0x1000):
[RID#1162] Server [NULL] resolution failed: [5]: Input/output error
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x0400):
[RID#1162] Failed to connect to server, but ignore mark offline is enabled.
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [sdap_id_op_connect_done] (0x4000):
[RID#1162] notify error to op #1: 5 [Input/output error]
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_dom_offline] (0x1000): [RID#1162]
Marking subdomain
USERS.AD2.COM offline
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [be_mark_subdom_offline] (0x1000):
[RID#1162] Marking subdomain
USERS.AD2.COM as inactive
* (2023-06-07 16:25:55): [be[ipa.ad1.com]] [ipa_srv_ad_acct_lookup_done] (0x0040):
[RID#1162] ipa_get_*_acct request failed: [1432158276]: Subdomain is inactive.
********************** BACKTRACE DUMP ENDS HERE *********************************
As far as I can tell the
users.ad2.com domain is "Active", but the bind fails to
the domain which forces a domain check to take place. Ultimately, this fails and the
domain is flagged offline so the authentication fails.
There seems to be some odd lines where the
users.ad2.com validation is trying to connect
to servers from the
ad1.com domain and global catalog validation which is failing. Not
sure why this would be taking place.
This seems to culminate in the line
(2023-06-07 16:25:55): [be[auth.ssdis.loc]] [fo_resolve_service_send] (0x0020): [RID#1162]
No available servers for service 'sd_gc_AD2.COM'
I can see actions taking place but cannot determine why they are taking place. I can
kinit as a
users.ad2.com user successfully. Likewise I can getent users and groups
successfully and see the details correctly. Yet authentication and authorisation is
failing for SSH logins
Any help or guidance on resolving this problem would be appreciated.