Thanks for responding.

what you said was not exactly my situation, but it got me poking around and finally I got the configuration working.

I find this interesting item when looking at various sssd subcommands;

[root@spikerealmd02 ~]# sssctl domain-list
amer.dell.com
apac.dell.com
emea.dell.com
japn.dell.com
dell.com
apac.dell.com
emea.dell.com
japn.dell.com

Why are the remote subdomains duplicated?  

Ultimately, this was my problem.

When I clear my logs:

sssctl remove-logs

and II do a: getent passwd admjesse_chan  
which still fails.

via 'grep -il admjesse_chan sssd_*.log',  I find string only in sssd_amer.dell.com.log  and sssd_nss.log.

sssd_apac.dell.com.log does the usual site-awareness lookup and identified primary LDAP server optimally.  So this sssd_be subprocess is working correctly.

Yet this log file never receives a query request for 'admjesse_chan@apac.dell.com'.

sssd_nss.log gets a query request for admjesse_chan.  It knows about remote subdomains:

(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_set_plugin] (0x2000): CR #1753: Setting "User by name" plugin
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_send] (0x0400): CR #1753: New request 'User by name'
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_process_input] (0x0400): CR #1753: Parsing input name [admjesse_chan]
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain apac.dell.com is Active
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain emea.dell.com is Active
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_domain_get_state] (0x1000): Domain japn.dell.com is Active

Since admjesse_chan was specified with domain, it performs a multi-domain search.  it searches local domain first.  It searches cache for admjesse_chan@amer.dell.com and then data_provider first:

(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_parse_name_for_domains] (0x0200): name 'admjesse_chan' matched without domain, user is admjesse_chan
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_set_name] (0x0400): CR #1753: Setting name [admjesse_chan]
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_select_domains] (0x0400): CR #1753: Performing a multi-domain search
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_search_domains] (0x0400): CR #1753: Search will check the cache and check the data provider
...

(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #1753: Object [admjesse_chan@amer.dell.com] was not found in cache
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR #1753: Looking up [admjesse_chan@amer.dell.com] in data provider

It dispatches to data_provider (child process sssd_be for amer.dell.com) and receives successful response back (no records found):
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [amer.dell.com][0x1][BE_REQ_USER][name=admjesse_chan@amer.dell.com:-]
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sbus_add_timeout] (0x2000): 0x55f844a5cac0
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_dp_internal_get_send] (0x0400): Entering request [0x55f842febb50:1:admjesse_chan@amer.dell.com@amer.dell.com]
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sbus_remove_timeout] (0x2000): 0x55f844a5cac0
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): dbus conn: 0x55f844a330f0
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 0 errno: 0 error message: Success


So next, it searches the next domain -- emea.dell.com:

(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400): Creating request for [emea.dell.com][0x1][BE_REQ_USER][name=admjesse_chan@emea.dell.com:-]
...
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sbus_dispatch] (0x4000): Dispatching.
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [sss_dp_get_reply] (0x1000): Got reply from Data Provider - DP error code: 3 errno: 5 error message: Input/output error
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_common_dp_recv] (0x0040): CR #1753: Data Provider Error: 3, 5, Input/output error
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_common_dp_recv] (0x0400): CR #1753: Due to an error we will return cached data
(Wed Jul  4 01:21:34 2018) [sssd[nss]] [cache_req_search_cache] (0x0400): CR #1753: Looking up [admjesse_chan@emea.dell.com] in cache

It catches the same I/O error when dispatching to the other data_providers:  apac.dell.com, japn.dell.com

So I focused on -- why are there duplicate entries in  sssctl domain-list?  Might it be trying to dispatch to some "ghost" data_provider (non-existent sssd_be child process)?  Thus I focused on my sssd.conf file.

i realized I don't need this line:

[sssd]
debug_level = 6
domains = amer.dell.com,apac.dell.com,emea.dell.com,japn.dell.com
domain_resolution_order = amer.dell.com, emea.dell.com, apac.dell.com, japn.dell.com

As without it, it will search domains in the order listed in "domains" line.

Tried that fix, made no difference.  Next I realized that https://lists.fedorahosted.org/pipermail/sssd-users/2015-February/002648.html
doesn't have these lines in each of his [domain/XXX] stanzas.


So I removed that line from each [domain/XXX] stanza  and did a:
sssctl cache-remove  (which restarts sssd service)

Now sssctl domain-list  shows me no dups:
[root@spikerealmd02 sssd]# sssctl domain-list

Now the dispatches out of sssd_nss.log to the various data_providers succeed.  And I see remote users:

[root@spikerealmd02 sssd]# id admjesse_chan
uid=525641(admjesse_chan@apac.dell.com) gid=525641(admjesse_chan@apac.dell.com) groups=525641(admjesse_chan@apac.dell.com),1008(apacunixusers@apac.dell.com),1000(apaclinuxeng@apac.dell.com),1001(apaclinuxsup@apac.dell.com)

I still don't see all expected groups for these users, so my ldap_use_tokengroups is still not 100% right.  Or maybe these missing groups are UNIVERSAL groups and I should be additionally querying dell.com.  Oh, well -problem for another day.  At least now x-subdomain auth is again working and tokengroups is (mostly) working.

Spike

On Tue, Jul 3, 2018 at 12:21 AM Sumit Bose <sbose@redhat.com> wrote:
On Mon, Jul 02, 2018 at 09:52:17AM -0500, Spike White wrote:
> Thanks for prompt reply.
>
> Yes, user name is strewn throughout sssd_nss.log.    Here's the last little
> bit:
>
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: Checking negative cache for [admjesse_chan@japn.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000):
> Checking negative cache for [NCE/USER/
> japn.dell.com/admjesse_chan@japn.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: [admjesse_chan@japn.dell.com] is not present in negative cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@japn.dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@japn.dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR
> #1653: Looking up [admjesse_chan@japn.dell.com] in data provider
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400):
> Issuing request for [0x55f842febb50:1:admjesse_chan@japn.dell.com@
> japn.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400):
> Creating request for [japn.dell.com
> ][0x1][BE_REQ_USER][name=admjesse_chan@japn.dell.com:-]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] (0x0400):
> Entering request [0x55f842febb50:1:admjesse_chan@japn.dell.com@japn.dell.com
> ]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400):
> Deleting request: [0x55f842febb50:1:admjesse_chan@apac.dell.com@
> apac.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@japn.dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@japn.dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_send] (0x0400): CR
> #1653: Looking up admjesse_chan@dell.com
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: Checking negative cache for [admjesse_chan@dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000):
> Checking negative cache for [NCE/USER/dell.com/admjesse_chan@dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: [admjesse_chan@dell.com] is not present in negative cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR
> #1653: Looking up [admjesse_chan@dell.com] in data provider
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400):
> Issuing request for [0x55f842febb50:1:admjesse_chan@dell.com@dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400):
> Creating request for [dell.com
> ][0x1][BE_REQ_USER][name=admjesse_chan@dell.com:-]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] (0x0400):
> Entering request [0x55f842febb50:1:admjesse_chan@dell.com@dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400):
> Deleting request: [0x55f842febb50:1:admjesse_chan@japn.dell.com@
> japn.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400):
> Deleting request: [0x55f842febb50:1:admjesse_chan@dell.com@dell.com]
>
> Specifically, this user is in apac.dell.com, so I want to make sure it's
> querying that subdomain.  It is.
>
> [root@spikerealmd02 sssd]# grep 'admjesse_chan.*apac' sssd_nss.log
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_send] (0x0400): CR
> #1653: Looking up admjesse_chan@apac.dell.com
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: Checking negative cache for [admjesse_chan@apac.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_ncache_check_str] (0x2000):
> Checking negative cache for [NCE/USER/
> apac.dell.com/admjesse_chan@apac.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_ncache] (0x0400):
> CR #1653: [admjesse_chan@apac.dell.com] is not present in negative cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@apac.dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@apac.dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_dp] (0x0400): CR
> #1653: Looking up [admjesse_chan@apac.dell.com] in data provider
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_issue_request] (0x0400):
> Issuing request for [0x55f842febb50:1:admjesse_chan@apac.dell.com@
> apac.dell.com]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_get_account_msg] (0x0400):
> Creating request for [apac.dell.com
> ][0x1][BE_REQ_USER][name=admjesse_chan@apac.dell.com:-]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_internal_get_send] (0x0400):
> Entering request [0x55f842febb50:1:admjesse_chan@apac.dell.com@apac.dell.com
> ]
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Looking up [admjesse_chan@apac.dell.com] in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [cache_req_search_cache] (0x0400):
> CR #1653: Object [admjesse_chan@apac.dell.com] was not found in cache
> (Sun Jul  1 15:21:12 2018) [sssd[nss]] [sss_dp_req_destructor] (0x0400):
> Deleting request: [0x55f842febb50:1:admjesse_chan@apac.dell.com@
> apac.dell.com]
>
> Yes, sss is in /etc/nsswitch.conf:
>
> [root@spikerealmd02 sssd]# grep sss /etc/nsswitch.conf
> passwd:     files sss
> shadow:     files sss
> group:      files sss
> #initgroups: files sss
> #netgroup:   files sss
>
> But I knew that already, as users in the local domain get resolved fine:
>
> [root@spikerealmd02 sssd]# id admspike_white
> uid=2025431(admspike_white@amer.dell.com) gid=2025431(
> admspike_white@amer.dell.com) groups=2025431(admspike_white@amer.dell.com
> ),1002(amerlinuxeng@amer.dell.com),1041(linux-core-engineering@amer.dell.com
> )
>
> Curiously, I see these local account lookups both in sssd_nss.log and
> in sssd_amer.dell.com.log  (AMER.DELL.COM is the local domain).  But for
> that   APAC user, I see the entries (above) in  sssd_nss.log, but no equiv
> entries in  sssd_apac.dell.com.log.

You should check what happens in teh sssd_apac log around the '(Sun Jul
1 15:21:12 2018)' timestamp. Maybe it is offline ('sssctl domain-status
apac.dell.com.log' might show this as well). There should be a debug
message with an offline error-code in sssd_nss.log as well.

HTH

bye,
Sumit
>
> Here's the systemctl status for sssd:
>
> [root@spikerealmd02 sssd]# systemctl status sssd
> ● sssd.service - System Security Services Daemon
>    Loaded: loaded (/usr/lib/systemd/system/sssd.service; enabled; vendor
> preset: disabled)
>    Active: active (running) since Mon 2018-06-25 00:05:10 CDT; 1 weeks 0
> days ago
>  Main PID: 82591 (sssd)
>    CGroup: /system.slice/sssd.service
>            ├─82591 /usr/sbin/sssd -i --logger=files
>            ├─82592 /usr/libexec/sssd/sssd_be --domain amer.dell.com --uid 0
> --gid 0 --logger=files
>            ├─82593 /usr/libexec/sssd/sssd_be --domain apac.dell.com --uid 0
> --gid 0 --logger=files
>            ├─82594 /usr/libexec/sssd/sssd_be --domain emea.dell.com --uid 0
> --gid 0 --logger=files
>            ├─82595 /usr/libexec/sssd/sssd_be --domain japn.dell.com --uid 0
> --gid 0 --logger=files
>            ├─82596 /usr/libexec/sssd/sssd_nss --uid 0 --gid 0 --logger=files
>            └─82597 /usr/libexec/sssd/sssd_pam --uid 0 --gid 0 --logger=files
>
> It's as if sssd_nss is properly invoking sub sssd process 82592 for
> amer.dell.com lookups.  But not invoking sub sssd process 82593 for
> apac.dell.com lookups.
>
> Spike
>
>
> On Mon, Jul 2, 2018 at 2:32 AM Sumit Bose <sbose@redhat.com> wrote:
>
> > On Sun, Jul 01, 2018 at 03:25:26PM -0500, Spike White wrote:
> > > sssd subject matter experts,
> > >
> > > Why is my sssd deployment not doing cross-subdomain AD authentication?
> > >
> > >
> > >
> > > *Background:*
> > >
> > > I have a parent AD domain DELL.COM with trusted subdomains AMER.DELL.COM
> > ,
> > > APAC.DELL.COM, EMEA.DELL.COM and JAPN.DELL.COM  Each subdomain has a
> > > transitive trust with DELL.COM.
> > >
> > > So all subdomains trust each other.
> > >
> > > I set up a first test VM deployment using sssd.  I set up the cross
> > > subdomain auth as in:
> > >
> > >
> > https://lists.fedorahosted.org/pipermail/sssd-users/2015-February/002648.html
> > >
> > >  It worked great – allowed cross subdomain authentication.  The only
> > thing
> > > it would not do was use tokengroups.  That is, the VM was fully
> > functional,
> > > but I had to add ‘ldap_use_tokengroups = false’  to the sssd.conf file.
> > >
> > > My AD experts have advised me that ‘tokengroups’ are an important AD
> > > optimization and I should use them, if at all possible.
> > >
> > > Using ldapsearch, I was able to verify that machine account didn’t have
> > the
> > > necessary privileges to query a user’s tokengroups.  Thus, the fault was
> > > mine – that this first sssd deployment couldn’t use tokengroups.
> > >
> > > So I did another sssd deployment, using another test VM.  Apparently, I
> > did
> > > the realm join command correct this time, as it’s able to use
> > tokengroups.
> > >
> > > BUT!  This second test VM is not allowing cross subdomain authentication
> > > and login.    How do I fix this so that I have use of both tokengroups
> > and
> > > cross subdomain authentication?
> > >
> >
> > ...
> >
> > >
> > > If I do the same query on the bad VM (spikerealmd02):
> > >
> > >
> > >
> > > [root@spikerealmd02 sssd]# id admjesse_chan
> > >
> > > id: admjesse_chan: no such user
> >
> > How do you know that tokengroups are working if the request fails and
> > the lookup is not recorded in the logs?
> >
> > >
> > >
> > >
> > > and I see nothing in the sssd_apac.dell.com.log file:
> > >
> > >
> > >
> > > [root@spikerealmd02 sssd]# grep -i admjesse_chan sssd_apac.dell.com.log
> > >
> > > [root@spikerealmd02 sssd]#
> >
> > Do you see the user name in sssd_nss.log? If not, is 'sss' listed in the
> > passwd line of /etc/nsswitch.conf?
> >
> > bye,
> > Sumit
> >
> > >
> > >
> > >
> > > Please help,
> > >
> > > Spike
> >
> > > _______________________________________________
> > > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> > > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > > List Archives:
> > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/HHNJ6AMXTSKF4UW3PLS6Y5MY5ADF6VCV/
> > _______________________________________________
> > sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> > To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> > Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> > List Archives:
> > https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/BASM657T2U6AVKBKVBQVJQQ7PNVU7UC7/
> >

> _______________________________________________
> sssd-users mailing list -- sssd-users@lists.fedorahosted.org
> To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/BKO5ZSU5AORKKVINJS4O6CZHRDXSQOF3/
_______________________________________________
sssd-users mailing list -- sssd-users@lists.fedorahosted.org
To unsubscribe send an email to sssd-users-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/sssd-users@lists.fedorahosted.org/message/X43BWGUCFVNLTAT346CQCPJZZ5QUGX2Z/