https://bugzilla.redhat.com/show_bug.cgi?id=1984591 not understanding the nature of the sssd bug introduced recently…
by Spike White
All (but particularly Sumit since he wrote the comments on
https://bugzilla.redhat.com/show_bug.cgi?id=1984591),
There are at least two problems created by this recently-introduced sssd
bug. One problem is solvable by the suggested work-around, the other is
not. The work-around suggested is:
[domain/name.of.joined.domain]
ad_enabled_domains = dom1.example.com, dom2.example.com,
dom3.example.com
In order to query only the desired AD domains.
What is the bug?
the sssd-ad man page says "The AD provider can be used to get user
information and authenticate users from trusted domains. Currently
only trusted domains in the same forest are recognized.".
What is happening is that untrusted AD domains are being discovered. A
very specific type of untrusted domains. When the joined domain has no
trust with that other domain, but that other domain trusts the original
domain – that is a one-way trust (the wrong way). To the joined domain,
this is an untrusted domain and should not be discovered.
This is actually very common in corporate environments.
You may have a main AD domain, call it CORP.COMPANY.COM. Then for testing
and new production evaluation, you might have a test AD domain called
LAB-TEST.COMPANY.COM. CORP.COMPANY.COM is tightly controlled, with full
audits and corporate security. LAB-TEST.COMPANY.COM is a test AD domain –
it’s the wild, wild west!
So LAB-TEST.COMPANY.COM trusts the main AD domain (in order that users can
log into this test domain with their CORP accounts). But CORP.COMPANY.COM
does not trust LAB-TEST.COMPANY.COM – nor should it!! (That’s the wild,
wild west, doing so would compromise corporate security.)
Thus, a server joined to domain CORP.COMPANY.COM should discover
CORP.COMPANY.COM and any domains trusted by CORP.COMPANY.COM. It should
*NOT* discover LAB-TEST.COMPANY.COM, as CORP.COMPANY.COM does not trust
this domain.
A server joined to LAB-TEST.COMPANY.COM should discover LAB-TEST.COMPANY.COM
and all domains trusted by LAB-TEST.COMPANY.COM. Including CORP.COMPANY.COM,
as LAB-TEST.COMPANY.COM trusts CORP.COMPANY.COM.
The bug is that a server joined to CORP.COMPANY.COM discovers
LAB-TEST.COMPANY.COM, which it shouldn’t.
What problems does this cause?
Two problems.
1. Many of these untrusted discovered “lab” domains are accessible
only to specific network locations. That is, they’re firewalled off to a
particular lab. So sssd attempts to query these inaccessible AD domains
and takes a long time to time out. This problem can be worked around by
the suggested work-around in the Bugzilla:
[domain/corp.company.com]
ad_enabled_domains = corp.company.com
So then, while LAB-TEST.COMPANY.COM is still erroneously discovered, it is
no longer searched. Sssd is again fast.
2. Bogus messages in /var/log/sssd_nss.log file. Even with no debug
level set in the [nss] stanza, these error messages appear multiple times a
second. It quickly fills up the /var/log filesystem.
[root@auspdfdlobv01 sssd]# cat sssd_nss.log |grep "The Data Provider
returned an error"
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
From debug level 9, it is clear that this is arising from a query of these
erroneously-discovered untrusted domains. Here’s an example of one
instance of above with debug level 9 turned on. So
emeaicmd.geodll.company.com is one of these erroneously-discovered
untrusted lab domains, that happens to be firewalled off from this
particular AD client:
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x1000): Got reply from
Data Provider - DP error code: 0 errno: 0 error message: Success
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9:
Looking up [oracle(a)company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9:
Object [oracle(a)company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain]
(0x0400): CR #9: Adding [oracle(a)company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User
oracle(a)company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding
[NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #9: Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR
#9: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #9:
Looking up oracle(a)emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9:
Checking negative cache for [oracle(a)emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking
negative cache for [NCE/USER/
EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #9: [
oracle(a)emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9:
Looking up [oracle(a)emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #9:
Object [oracle(a)emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #9: Looking
up [oracle(a)emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_get_account_msg] (0x0400): Creating
request for [EMEAICMD.geodll.company.com
][0x3][BE_REQ_INITGROUPS][name=oracle@emeaicmd.geodll.company.com:-]
(2021-10-07 9:50:02): [nss] [sbus_add_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sss_dp_internal_get_send] (0x0400): Entering
request [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12:
Looking up [oracle(a)company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12:
Object [oracle(a)company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache_add_to_domain]
(0x0400): CR #12: Adding [oracle(a)company.com] to negative cache
(2021-10-07 9:50:02): [nss] [is_user_local_by_name] (0x0400): User
oracle(a)company.com is a local user
(2021-10-07 9:50:02): [nss] [sss_ncache_set_str] (0x0400): Adding
[NCE/USER/company.com/oracle@company.com] to negative cache
(2021-10-07 9:50:02): [nss] [cache_req_validate_domain_type] (0x2000):
Request type POSIX-only for domain EMEAICMD.geodll.company.com type POSIX
is valid
(2021-10-07 9:50:02): [nss] [cache_req_set_domain] (0x0400): CR #12: Using
domain [EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_prepare_domain_data] (0x0400): CR
#12: Preparing input data for domain [EMEAICMD.geodll.company.com] rules
(2021-10-07 9:50:02): [nss] [cache_req_search_send] (0x0400): CR #12:
Looking up oracle(a)emeaicmd.geodll.company.com
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12:
Checking negative cache for [oracle(a)emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_ncache_check_str] (0x2000): Checking
negative cache for [NCE/USER/
EMEAICMD.geodll.company.com/oracle@emeaicmd.geodll.company.com]
(2021-10-07 9:50:02): [nss] [cache_req_search_ncache] (0x0400): CR #12: [
oracle(a)emeaicmd.geodll.company.com] is not present in negative cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12:
Looking up [oracle(a)emeaicmd.geodll.company.com] in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_cache] (0x0400): CR #12:
Object [oracle(a)emeaicmd.geodll.company.com] was not found in cache
(2021-10-07 9:50:02): [nss] [cache_req_search_dp] (0x0400): CR #12:
Looking up [oracle(a)emeaicmd.geodll.company.com] in data provider
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Issuing
request for [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_issue_request] (0x0400): Identical
request in progress: [0x564d6be36a70:3:oracle@emeaicmd.geodll.company.com@
EMEAICMD.geodll.company.com]
(2021-10-07 9:50:02): [nss] [sss_dp_req_destructor] (0x0400): Deleting
request: [0x564d6be36a70:3:oracle@company.com@company.com]
(2021-10-07 9:50:02): [nss] [sbus_remove_timeout] (0x2000): 0x564d6ccd6670
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): dbus conn:
0x564d6ccc9300
(2021-10-07 9:50:02): [nss] [sbus_dispatch] (0x4000): Dispatching.
(2021-10-07 9:50:02): [nss] [sss_dp_get_reply] (0x0010): The Data Provider
returned an error [org.freedesktop.DBus.Error.Failed]
The suggested work-around does not resolve problem #2.
BTW, here is a listing of the domains discovered on that sssd client:
[root@auspdfdlobv01 ~]# sssctl domain-list
amer.company.com
company.com
japn.company.com
emea.company.com
apac.company.com
EMEAICMD.geodll.company.com
geodll.company.com
EMEAICM.GEODLL.COMPANY.COM
alienware.com
corp.svcs
perotsystems.net
companyservices.dmz
Beer.Town
production.online.company.com
jp-poclab.companypoc.com
emea-poclab.companypoc.com
oldev.preol.company.com
olqa.preol.company.com
ap-poclab.companypoc.com
[root@auspdfdlobv01 ~]#
This sssd client is joined to amer.company.com, so the only trusted domains
are the first 5. The parent domain and the 4 regional domains. All
those other domains below that are untrusted domains. More specifically,
they trust company.com, but company.com does not trust them. (one way
trust – the wrong way.) Some look like the real wild wild west (Beer.Town
?).
Spike
2 years, 6 months
sss_ssh_authorizedkeys returning "Error looking up public keys"
by Philip Colmer
We are using Ubuntu 18.04 with sssd 1.16.1-1ubuntu1.8.
sssd is being used to allow users to sign in with SSH keys matched against keys stored in OpenLDAP. This has been used for several years but, just recently, a couple of users were reporting that they could not SSH to the server. Further investigation showed that running "sss_ssh_authorizedkeys" for these specific users returned "Error looking up public keys". The tool continues to work correctly for other users.
I cannot find any information that might suggest why the tool is not managing to get the keys for these users.
I have tried building the code from the GitHub source but it errors out with this:
In file included from ./src/util/sss_pam_data.h:33:0,
from src/util/sss_pam_data.c:27:
./src/util/debug.h:92:44: error: unknown type name ‘uid_t’; did you mean ‘__id_t’?
int chown_debug_file(const char *filename, uid_t uid, gid_t gid);
^~~~~
__id_t
./src/util/debug.h:92:55: error: unknown type name ‘gid_t’; did you mean ‘__id_t’?
int chown_debug_file(const char *filename, uid_t uid, gid_t gid);
^~~~~
__id_t
Makefile:22359: recipe for target 'src/util/libsss_util_la-sss_pam_data.lo' failed
Can anyone suggest how I can troubleshoot this tool in order to fix the error or how I can get it to build on Ubuntu?
Thanks.
Philip
2 years, 6 months
Trouble-shooting sssd’s ‘Automatic Kerberos Host Keytab Renewal’ with AD back-end….
by Spike White
Sssd experts,
*Short summary: * How can we troubleshoot sssd’s ‘Automatic Kerberos Host
Keytab Renewal’ process? We have ~0.4% of our Linux servers dropping
off the AD domain monthly.
*Longer explanation:*
Over the past two years, we have on-boarded sssd as our Linux AD
integration component. Largely displacing a former commercial product that
did the same.
We have about ~20K Linux servers that are sssd-enabled. A mix of RHEL6,
RHEL7, RHEL8, Oracle Linux 6, 7 and 8. We have ~7K Linux servers still on
the old commercial product. (For certain edge-case scenarios, such as
DMZs, the commercial product works better.)
Our AD forest is a single AD forest, with 4 regional child domains. All
with transitive trust. Sssd auto-discovers parent domain and all 4 child
domains, no problem – whenever it’s adcli joined to its regional local
domain.
Why are I writing this?
Because we are researching an ongoing problem reported by L1 server ops.
About 70 – 80 sssd-enabled Linux servers / month drop off the domain. Out
of our current sssd-enabled population of ~20K server, that’s not
horrible. But still it should be better. (Our former commercial product
did better.)
It’s not limited to one particular OS, OS version, build location or
region. We have surveyed; it seems to occur randomly among all OS
versions, regions and locations.
To be clear, it’s extremely likely that this behavior arising from some
subtle misconfiguration on our part – not from any sssd or adcli or
Kerberos bug. We have a couple of configuration improvements we’re
pursuing. (Kerberos max ticket lifetime mismatch between AD and
/etc/krb5.conf file for instance.)
We are taking sssd’s default settings for
ad_maximum_machine_account_password_age and
ad_machine_account_password_renewal_opts. So after 30 days, sssd will
attempt daily to renew the host Kerberos keytab file. It should re-attempt
daily if not renewed. By company policy, our AD disables any machine
accounts that have not renewed their credentials in 40 days. So when we
find servers that have dropped off the domain, it’s because they have not
renewed their AD machine accounts in 40 days.
We have SR’s open with our OS vendors (Redhat and Oracle respectively) for
months now. To no great help. (They gave a few suggestions, but none
panned out.)
We thought we were hitting this bug:
https://github.com/SSSD/sssd/issues/4762
But packet captures proved that adcli update is using TCP on RHEL7/8.
Thus, this might be a potential problem, but only on RHEL6. (BTW
‘udp_preference_limit = 0’ doesn’t force use of TCP for the kpasswd
invocation in RHEL6 – it still uses UDP. Thus, the recommended work-around
for this bug doesn’t work.)
So that isn’t our underlying problem.
We’re at a loss now – as you can see, we’re grasping at straws.
How can we troubleshoot sssd’s ‘automatic Kerberos Host keytab renewal’
process? Whenever we inspect a particular server it works. We can’t run
all sssd clients at debug level 9; it fills up /var/log filesystem after a
few days of that. We’re interested in troubleshooting that one particular
sssd process on all clients; not all parts of sssd.
Other than a steep learning curve (on our part), obscure situations (like
DMZ auto-discovery of AD controllers) and exotic scenarios (like above),
we’re quite happy with our 2 yr journey of direct AD integration with
sssd. Obviously, the troubleshooting tools on RHEL6 are very minimal.
But certainly, overall the quality of sssd on RHEL7/8 is excellent. AD
integration has innumerable devils in the details; I’m amazed that sssd
performs as well as it does against our multi-domain forest.
Spike
PS the problem with sssd auto-discovery of AD controllers in DMZs has been
fixed in a recent sssd release. The better discovery algorithm was
implemented – same one used by Windows clients and commercial products.
It’s just that recent sssd version is not on RHEL7 or 8.
2 years, 6 months
Any way to auto create groups while still mapping UIDs to uidNumber?
by Kurt Stine
I was told this would be a better place than github issues.
We're moving from an ldap environment to an AD environment. This means we have a large amount of users who are still linked with their original ldap UIDs. Unfortunately, changing the UIDs to the automatically assigned AD UIDs is not an option. Is there a way for SSSD to both inherit a UID if it exists or create its own if it doesn't? Also, is it possible to sync AD groups as well?
Right now we have this:
#ldap_id_mapping = True
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
Unfortunately, even removing the last line doesn't sync AD groups. Are there any options to sync them?
Thanks,
Kurt
2 years, 6 months