Yes, this IPA group was always missing in the 1st IPA server, and another IPA group was missing in the 2nd IPA server.
After executing systemctl stop sssd ; sss_cache -E ; rm -rf /var/lib/sss/db/* ; systemctl start sssd on each server I can see both IPA groups on both servers.
I can't explain this, only executing sss_cache -E the problem persisted, I had to delete cache files too, and to restart services.

Then I tried to do the same procedure several times, but it seems that I can see always correct groups doing a id username@mydomain.com command.
I tried to do a ssh connection using my AD credential on all my IPA clients and now it works.

So, do you think the issue is solved now?
Only a cache problem after the update of the release of the two servers?

Thanks, bye

Il giorno ven 8 mar 2019 alle ore 11:50 Sumit Bose <sbose@redhat.com> ha scritto:
On Thu, Mar 07, 2019 at 04:10:09PM +0100, Morgan Marodin wrote:
> Another strange behaviour ...
>
> From 1st IPA server:
>
>
> *[root@mlv-ipa01 ~]# id morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>uid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>) gid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>)
> groups=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>),1147803679(aaa local proxy
> exclusion@mydomain.com <exclusion@mydomain.com>),1147802790(mysite
> signature mycompany@mydomain.com <mycompany@mydomain.com>),1147801237(mysite
> users@mydomain.com <users@mydomain.com>),1147862761(ipa admins@mydomain.com
> <admins@mydomain.com>),1147801336(mysite it@mydomain.com
> <it@mydomain.com>),1147800813(domain users@mydomain.com
> <users@mydomain.com>)*

Getting the group memberships of a user is a multi step process and SSSD
on the IPA server currently saves the memberships it got for the user
from AD into the cache before it tries to lookup the memberships in IPA
groups from the IPA server. If there is an issue during this step the
user will only have the AD memberships.

Are the IPA groups always missing on this server? If there already is a
higher debug_level set in the [domain/..] section on this server please
send the SSSD logs. If not please add debug_level=9 at least to the
[domain/...] section and restart SSSD. The run the id command again on
the server. If the IPA group is still missing please send the logs
directly. Otherwise please wait until the group got lost.

You might be able to trigger the loss by invalidation the cache with
'sss_cache -E' and running lookups for this group with 'getent group
ad_ipa_admins' or id lookups for other member of this groups.

bye,
Sumit

> From 2nd IPA server:
>
>
> *[root@mlv-ipa02 ~]# id morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>uid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>) gid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>)
> groups=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>),1147803679(aaa local proxy
> exclusion@mydomain.com <exclusion@mydomain.com>),1147802790(mysite
> signature mycompany@mydomain.com <mycompany@mydomain.com>),1147801237(mysite
> users@mydomain.com <users@mydomain.com>),1147862761(ipa admins@mydomain.com
> <admins@mydomain.com>),1147801336(mysite it@mydomain.com
> <it@mydomain.com>),1147800813(domain users@mydomain.com
> <users@mydomain.com>),217403007(ad_ipa_admins)*
> From the test IPA client:
>
>
> *[root@mlv-testipa01 ~]# id morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com> uid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>) gid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>)
> groups=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>),1147803679(aaa local proxy
> exclusion@mydomain.com <exclusion@mydomain.com>),1147802790(mysite
> signature mycompany@mydomain.com <mycompany@mydomain.com>),1147801237(mysite
> users@mydomain.com <users@mydomain.com>),1147862761(ipa admins@mydomain.com
> <admins@mydomain.com>),1147801336(mysite it@mydomain.com
> <it@mydomain.com>),1147800813(domain users@mydomain.com
> <users@mydomain.com>),217403007(ad_ipa_admins)*
> From another IPA client (with the same issue):
>
> *[root@mlv-box02 ~]# id morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>uid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>) gid=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>)
> groups=1143802726(morgan.marodin@mydomain.com
> <morgan.marodin@mydomain.com>),1147803679(aaa local proxy
> exclusion@mydomain.com <exclusion@mydomain.com>),1147802790(mysite
> signature mycompany@mydomain.com <mycompany@mydomain.com>),1147801237(mysite
> users@mydomain.com <users@mydomain.com>),1147862761(ipa admins@mydomain.com
> <admins@mydomain.com>),1147801336(mysite it@mydomain.com
> <it@mydomain.com>),1147800813(domain users@mydomain.com
> <users@mydomain.com>)*
>
> As you can see the *217403007(ad_ipa_admins)* is given back only from some
> server, not all, and the match is done via HBAC in that group.
>
> Bye
>
> Il giorno gio 7 mar 2019 alle ore 15:38 Morgan Marodin <morgan@marodin.it>
> ha scritto:
>
> > Hi.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[root@mlv-testipa01 ~]# host -t SRV _kerberos._udp.IPA.MYDOMAIN.COM
> > <http://udp.IPA.MYDOMAIN.COM>_kerberos._udp.IPA.MYDOMAIN.COM
> > <http://udp.IPA.MYDOMAIN.COM> has SRV record 0 100 88
> > mlv-ipa01.ipa.mydomain.com
> > <http://mlv-ipa01.ipa.mydomain.com>._kerberos._udp.IPA.MYDOMAIN.COM
> > <http://udp.IPA.MYDOMAIN.COM> has SRV record 0 100 88
> > mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com>.[root@mlv-testipa01 ~]# host -t SRV
> > _kerberos._tcp.IPA.MYDOMAIN.COM
> > <http://tcp.IPA.MYDOMAIN.COM>_kerberos._tcp.IPA.MYDOMAIN.COM
> > <http://tcp.IPA.MYDOMAIN.COM> has SRV record 0 100 88
> > mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com>._kerberos._tcp.IPA.MYDOMAIN.COM
> > <http://tcp.IPA.MYDOMAIN.COM> has SRV record 0 100 88
> > mlv-ipa01.ipa.mydomain.com
> > <http://mlv-ipa01.ipa.mydomain.com>.[root@mlv-testipa01 ~]# host -t any
> > mlv-ipa01.ipa.mydomain.com
> > <http://mlv-ipa01.ipa.mydomain.com>mlv-ipa01.ipa.mydomain.com
> > <http://mlv-ipa01.ipa.mydomain.com> has address
> > 192.168.0.65[root@mlv-testipa01 ~]# host -t any mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com>mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com> has address
> > 192.168.0.66mlv-ipa02.ipa.mydomain.com <http://mlv-ipa02.ipa.mydomain.com>
> > has SSHFP record 1 1
> > 1A1F9D66E9B156AA14A6739C46252814163D7DB2mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com> has SSHFP record 1 2
> > 7BCDACE8C28B624E938D646791734C740F0833A0221E1B573D323EC4
> > A726FB07mlv-ipa02.ipa.mydomain.com <http://mlv-ipa02.ipa.mydomain.com> has
> > SSHFP record 3 2 BE4850443507A7B6E174576ACDAEBA6D3839C9476BFBAA09B0753D75
> > F7C62DC1mlv-ipa02.ipa.mydomain.com <http://mlv-ipa02.ipa.mydomain.com> has
> > SSHFP record 3 1 32ADAF67104C0A6ADF94446B3BCB5235550C9D14*
> >
> > And this is the version and the *sssd.conf* of the test client:
> >
> >
> > *[root@mlv-testipa01 ~]# sssd --version1.16.2*
> >
> > *[root@mlv-testipa01 ~]# cat /etc/sssd/sssd.conf*
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > *[domain/ipa.mydomain.com <http://ipa.mydomain.com>]debug_level =
> > 9cache_credentials = Truekrb5_store_password_if_offline = Trueipa_domain =
> > ipa.mydomain.com <http://ipa.mydomain.com>id_provider = ipaauth_provider =
> > ipaaccess_provider = ipaipa_hostname = mlv-testipa01.ipa.mydomain.com
> > <http://mlv-testipa01.ipa.mydomain.com>chpass_provider = ipaipa_server =
> > _srv_, mlv-ipa02.ipa.mydomain.com
> > <http://mlv-ipa02.ipa.mydomain.com>ldap_tls_cacert =
> > /etc/ipa/ca.crt#dns_discovery_domain = mysite._sites.mydomain.com
> > <http://sites.mydomain.com>[sssd]services = nss, sudo, pam, sshdomains =
> > ipa.mydomain.com <http://ipa.mydomain.com>[nss]homedir_substring =
> > /homeoverride_shell =
> > /bin/bash[pam][sudo][autofs][ssh][pac][ifp][secrets][session_recording]*
> >
> > Any other check regarding DNS?
> > Thanks, bye
> >
> > Il giorno gio 7 mar 2019 alle ore 12:13 Sumit Bose <sbose@redhat.com> ha
> > scritto:
> >
> >> On Thu, Mar 07, 2019 at 09:15:30AM +0100, Morgan Marodin wrote:
> >> > Sorry, could I try to delete and recreate the trust?
> >> > Or do you have other suggestions?
> >> >
> >> > Please let me know, thanks
> >> >
> >> > Il giorno ven 1 mar 2019 alle ore 15:13 Morgan Marodin <
> >> morgan@marodin.it>
> >> > ha scritto:
> >> >
> >> > > It's one of our local Domain Controller, in this site there is 2 DC.
> >> > > Could I try to delete and recreate the trust? Or do you have other
> >> > > suggestions?
> >>
> >> Hi,
> >>
> >> I think recreate the trust won't help. It looks like a DNS issue. Please
> >> check if
> >>
> >>     host -t SRV _kerberos._udp.IPA.MYDOMAIN.COM
> >>
> >> or
> >>
> >>     host -t SRV _kerberos._udp.IPA.MYDOMAIN.COM
> >>
> >> return this DC or any other AD DCs. Only IPA servers should be returned
> >> here.
> >>
> >> bye,
> >> Sumit
> >>
> >> > >
> >> > > Thanks
> >> > >
> >> > > Il giorno ven 1 mar 2019 alle ore 11:13 Sumit Bose <sbose@redhat.com>
> >> ha
> >> > > scritto:
> >> > >
> >> > >> On Fri, Mar 01, 2019 at 09:07:26AM +0100, Morgan Marodin wrote:
> >> > >> > Sorry, any news from my logs?
> >> > >>
> >> > >> sorry for the delay. What kind of server is 192.168.0.15? It looks
> >> like
> >> > >> the client assumes it is a KDC for IPA.MYDOMAIN.COM but it returns
> >> > >> 'Realm not local to KDC' when getting a request for this realm.
> >> > >> Additionally it looks like it cannot decrypt IPA-AD cross-realm
> >> tickets.
> >> > >>
> >> > >> bye,
> >> > >> Sumit
> >> > >>
> >> > >> >
> >> > >> > Please let me know, thanks.
> >> > >> > Morgan
> >> > >> >
> >> > >> > Il giorno mar 26 feb 2019 alle ore 11:56 Morgan Marodin <
> >> > >> morgan@marodin.it>
> >> > >> > ha scritto:
> >> > >> >
> >> > >> > > You can find attached a tar.gz file with the logs of the server
> >> and
> >> > >> the
> >> > >> > > testing client, captured after done a restart of the *sssd*
> >> daemon
> >> > >> and a *sss_cache
> >> > >> > > -E* command, on both parts.
> >> > >> > > I sanitized all logs, a long work!
> >> > >> > >
> >> > >> > > Il giorno lun 25 feb 2019 alle ore 17:14 Sumit Bose <
> >> sbose@redhat.com>
> >> > >> ha
> >> > >> > > scritto:
> >> > >> > >
> >> > >> > >> On Mon, Feb 25, 2019 at 03:55:52PM +0100, Morgan Marodin wrote:
> >> > >> > >> > The right HBAC is called *allow_ad_ipa_admins*, that match
> >> the IPA
> >> > >> group
> >> > >> > >> > *ad_ipa_admins*, that is trusted with the group '*IPA
> >> Admins*' in
> >> > >> Active
> >> > >> > >> > Directory.
> >> > >> > >> > I tested the *id morgan.marodin@mydomain.com <
> >> > >> > >> morgan.marodin@mydomain.com>*
> >> > >> > >> > command both in the client and the server, they differ only
> >> for
> >> > >> the last
> >> > >> > >> > part *,219402407(ad_ipa_admins)*.
> >> > >> > >> > I can see it in the client, not in the server.
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> That explains the changing behaviour on the client. The client
> >> gets
> >> > >> all
> >> > >> > >> group memberships from the server and it looks like the server
> >> once
> >> > >> in a
> >> > >> > >> while has issues to add the IPA group memberships to AD users.
> >> > >> > >>
> >> > >> > >> Please add now debug_level=9 to the [nss] and [domain/...]
> >> sections
> >> > >> on
> >> > >> > >> an IPA server and restart SSSD. Then please try to reproduce the
> >> > >> state
> >> > >> > >> where ad_ipa_admins is missing. For this you can try to restart
> >> SSSD
> >> > >> and
> >> > >> > >> call the id command afterwards or calling 'sss_cache -E' to
> >> > >> invalidate
> >> > >> > >> the cached data before calling id.
> >> > >> > >>
> >> > >> > >> If you change sssd.conf on the servers it would be helpful to
> >> see a
> >> > >> > >> (sanitized) version of sssd.conf as well.
> >> > >> > >>
> >> > >> > >>
> >> > >> > >> bye,
> >> > >> > >> Sumit
> >> > >> > >>
> >> > >> > >
> >> > >>
> >> > >
> >>
> >
>