Hi again,

I've tried quite a lot of things (including the very latest version 0.7.0 of gssproxy) but I keep failing to get it to work on fedora25.

So in a last ditch attempt I created a reproducer in the form of a vagrant script that creates a test environment that has the problem.

I've tested the vagrant script on fedora25 and using kvm/libvirt with all the latest patches applied.
vagrant versions :
vagrant-libvirt-doc-0.0.35-3.fc25.noarch
vagrant-1.8.5-2.fc25.noarch
vagrant-libvirt-0.0.35-3.fc25.noarch  <- this should be 0.0.35-3 or later

vagrant-libvirt must be up2date with the latest build or else it will fail to configure the network properly. (https://bugzilla.redhat.com/show_bug.cgi?id=1426565)

all passwords are 'centos73'

dns/ip info
ipa.example.test 192.168.122.2/24
nfs.example.test 192.168.122.3/24
fedoraclient.example.test 192.168.122.4/24

I hope you are willing to give this a try cause I'm kinda out of options here.

there should be a plain text attachment ;Vagrantfile' to this email, if not pm me and I will send it to you.

Rob Verduijn


I doubt that you are new to vagrant, butl some instructions anyway:

if you don't have the kvm/libvirt setup yet
dnf install -y @virtualization && systemctl enable libvirtd.service && systemctl start libvirtd.service

install vagrant :
dnf install -y vagrant

create a working dir and change to it
mkdir $HOME/vagrant
cd $HOME/vagrant

put the file 'Vagrantfile' in the directory

start the vagrant script
vagrant up

this command will take a (very) long time to finish.

Here is what it does (in parallel where possible)

vagrant will download the vagrant box images for fedora25 and centos73

create a virtual nat network 'vagrant_libvirt' 192.168.121.0/24 in which the dns and gateway are 192.168.121.1

all systems are dual-homed because vagrant requires a control/management interface that resides in the vagrant_libvirt network

the environment will use the virtual nat network 'default' the default subnet for that network is 192.168.122.0/24

install and configure ipa on centos73
add hosts, users and automount maps
and finally configure the firewall

configure a nfs server on centos73
and when it notices the ipa server is available:
join the ipa domain
create a principal for the kerberised nfs
import the principal to the local keytab

configure a fedora25 ipa client
join the ipa domain
configure automounts
install apache
create a principal for apache
configure gssproxy
restart services where needed

during the installation it is very likely the ipa server will starve for entrophy,
usually a lot of messages saying 'waiting for ipa configuration to finish' are a good hint more entrophy is needed.

scroll up to see if there is a entrophy message somewhere.

and if you see such a message
you can either wait a very long time for the entrophy to appear (really, it will appear)

or log in to the console of the ipa server via the virt-manager ( as not to interrupt the script unless you like to start over, see below how)
and generate some entrophy in the usual way (typing/drive access/etc/etc)

When the scipt is finished:

(The 'vagrant ssh' commands won't work untill the 'vagrant up' command is finished)

use the following command to log in to the fedoraclient:
vagrant ssh fedoraclient
then use sudo to become root (no password on sudo)

the nfs sercer
vagrant ssh nfsserver

the ipa server
vagrant ssh ipaserver

in case you interupted the 'vagrant up' reset everything by using the destroy command.

Or if you simply want to get rid of the vagrant machines.
vagrant destroy

the destroy command will not delete the box images, you will need to remove those manually if you want those gone.
they can be found in /var/lib/libvirt/images (which also contains any other vm images you might have created)



2017-01-29 17:31 GMT+01:00 Rob Verduijn <rob.verduijn@gmail.com>:
Hello,

Since you mentioned something about nfs-utils and changes I thought I try the latest nfs-utils from testing as well.
(whoa....major version upgrade coming up....lets see if I can find a changelog)

Anyway , still no go.
Normal users work fine though.

Rob Verduijn

2017-01-29 17:03 GMT+01:00 Rob Verduijn <rob.verduijn@gmail.com>:

I did another default install with centos7 applied all the patches, and made it a domain member, configured the automount (as above)

configured it the same way I described above with the minor difference that I used only 1 configuration file for gssproxy:
cat /etc/gssproxy/gssproxy.conf
[gssproxy]
debug = true
debug_level = 9

[service/HTTP]
  mechs = krb5
  cred_store = client_keytab:/etc/gssproxy/http.keytab
  cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
  euid = 48

[service/nfs-server]
  mechs = krb5
  socket = /run/gssproxy.sock
  cred_store = keytab:/etc/krb5.keytab
  trusted = yes
  kernel_nfsd = yes
  euid = 0

[service/nfs-client]
  mechs = krb5
  cred_store = keytab:/etc/krb5.keytab
  cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_%U
  cred_store = client_keytab:/var/lib/gssproxy/clients/%U.keytab
  cred_usage = initiate
  allow_any_uid = yes
  trusted = yes
  euid = 0


In the service/HTTP instance I changed keytab to client_keytab.
and tested it and it worked.

so I did an strace on rpc.gssd on centos 7.3
And got this :
epoll_wait(4,
{{EPOLLIN, {u32=9, u64=9}}}, 32, -1) = 1
read(9, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45
clone(child_stack=0x7fd125bfdeb0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7fd125bfe9d0, tls=0x7fd125bfe700, child_tidptr=0x7fd125bfe9d0) = 2531
epoll_wait(4,


on the fedora25-kde the strace looks like this :
epoll_wait(4,
[{EPOLLIN, {u32=12, u64=12}}], 32, -1) = 1
read(12, "mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45
clone(child_stack=0x7f75c4aa3ef0, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, parent_tidptr=0x7f75c4aa49d0, tls=0x7f75c4aa4700, child_tidptr=0x7f75c4aa49d0) = 16294
epoll_wait(4,

This is al more or less giberish to me, but I think the important bit is 'mech=krb5 uid=48 enctypes=18,17,"..., 32768) = 45'
and that is the same on both systems.


Also the environment of the rpc.gssd proc entry

the output from /proc/<pid>/environ on centos7.3:
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binRPCNFSDARGS= 8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=GSSDARGS=SVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes

The output from /proc/<pid>/environ on fedora25-kde:
LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/binJOURNAL_STREAM=8:22877RPCNFSDARGS= 8RPCMOUNTDARGS=STATDARGS=SMNOTIFYARGS=RPCIDMAPDARGS=RPCGSSDARGS=RPCSVCGSSDARGS=BLKMAPDARGS=GSS_USE_PROXY=yes

Rob Verduijn


2017-01-29 15:33 GMT+01:00 Rob Verduijn <rob.verduijn@gmail.com>:
Hi,

I guess I'm missing something very basic here.

Just to make sure I'm not missing something very elementary here, could you review the following ?

So all my exports work fine when using ordinary user accounts on fedora clients (kde-spin)
So I doubt that there is something wrong on the nfs server side or the ipa bit.
I installed the default fedora25-kde spin with additional groups : @domain-client
no other packages where added.

A host and dns entry were created in freeipa
An otp password is set

The command 'ipa-client install --enable-dns-updates -U -w ********'  has been applied
The command 'ipa-client-automount' has been issued.

a http principal has been created for this host via the freeipa gui
imported it to the client with
ipa-getkeytab -p HTTP/fedora25-kde.example.com@EXAMPLE.COM -k /var/lib/gssproxy/clients/httpd.keytab

created the 00-apache.conf
cat > /etc/gssproxy/00-apache.conf << EOF
[service/apache]
  mechs = krb5
  cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache
  cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab
  cred_usage = initiate
  euid = 48
EOF

added debug settings to gssproxy.conf
echo 'debug = true' >> /etc/gssproxy/gssproxy.conf
echo 'debug_level = 9' >> /etc/gssproxy/gssproxy.conf

nfsidmap resolving works better for me when the method is set to sss.
the command 'sed -i 's/#Method.*/Method=sss/' /etc/idmapd.conf'  is applied

updated everything :
dnf update -y

added the latest gssproxy from testing
dnf update -y gssproxy --enablerepo=updates-testing

add httpd
dnf install -y httpd

reboot

the automount map from ipa that I am using :
ipa automountkey-show
Location: default
Map: auto.direct
Key: /network/music
  Key: /network/music
  Mount information: -rw,noatime,timeo=14,soft,sec=krb5p,acl nfs.example.com:/music

login to the system as root
'ls /network/music' gives a directory listing

issue 'su - apache -s /bin/bash'
now 'ls /network/music' gives 'ls: cannot open directory '/network/music': Permission denied'

I logout
ctrl-d , ctrl-d

I login as an ipa domain user. (not admin)
'ls /network/music' gives a directory listing.


the root user does not exist in the domain
the apache user does not exits in the domain

the apache user has local id 48 on the nfs server and local id 48 on the nfs client.
the apache user has group apache with id 48 on the nfs client and nfs server.





2017-01-29 14:10 GMT+01:00 Simo Sorce <ssorce@redhat.com>:
So it turned out I was not reproducing the issue, I had an older version of gssproxy than I thought, and incorrect logging level in the config file.

Once I had time to sit down and fix my client by installing the latest updates-testing package and making sure the debug level was properly set all worked as expected for me on Fedora 25.

I was suspecting some issues with a relatively recent patchset that went into nfs-utils which switched from using fork() to use threads, but it turned out this situation is already properly handled in the gss plugin used to talk to gssproxy. (CCing Jeff who reviewed those patches so he knows it is all good for now :-)

Rob,
in order to find out what is going on you may want to try to strace rpc.gssd on the client system and try to figure out why gssproxy is not being used, also check if /proc/<pid>/env of rpc.gssd does in fact include the proper environment variable.

Simo.

----- Original Message -----
> From: "Simo Sorce" <simo@redhat.com>
> To: "The GSS-Proxy developers and users mailing list" <gss-proxy@lists.fedorahosted.org>
> Sent: Saturday, January 28, 2017 7:05:41 AM
> Subject: Re: [gssproxy] Re: gssproxy broken on fedora
>
> Hi Rob,
> I can reproduce your issue now and I am looking into this issue and I
> suspect nfs-utils may have broken gss-proxy support recently.
>
> I am not sure yet, but some critical areas have been changed, so I need
> to better investigate them. It will take me a little time as I am at
> Devconf.cz now, but hopefully I will have an answer for you in a few
> days.
>
> Simo.
>
> On Sat, 2017-01-21 at 13:10 +0100, Rob Verduijn wrote:
> > Another fresh install of a fedora25 client, now with the new gssproxy
> > 0.6.0 package, but that one does not work either.
> >
> > Created the file /etc/gssproxy/00-apache.conf
> > [service/apache]
> >   mechs = krb5
> >   cred_store = ccache:FILE:/var/lib/gssproxy/clients/krb5cc_apache
> >   cred_store = client_keytab:/var/lib/gssproxy/clients/httpd.keytab
> >   cred_usage = initiate
> >   euid = 48
> >
> > put the keytab in the specified place
> > added debug settings to /etc/gssproxy/gssproxy.conf
> > [gssproxy]
> > debug = true
> > debug_level = 9
> >
> > I did  not touch the file /etc/sysconfig/nfs
> > Since the default setting of GSS_USE_PROXY="yes" is what I want
> >
> > Checked the logs, nothing in there besides what you get from
> > systemctl status gssproxy.service
> >
> > su - apache -s /bin/bash
> > and no access to the shares is allowed
> >
> > Weird thing I noticed, when browsing as root the nfs mounts are
> > readable ( root is squashed ) but not as the apache user.
> > Root gets to read it with nobody:nobody privileges but apache with
> > apache:apache is refused.
> >
> > I'm really at a loss as to what to do next.
> >
> > Do I need to set an environment variable to make this work ?
> >
> > Rob Verduijn
> >
> > 2017-01-04 23:14 GMT+01:00 Rob Verduijn <rob.verduijn@gmail.com>:
> > >
> > > 2017-01-04 20:56 GMT+01:00 Simo Sorce <simo@redhat.com>:
> > > > On Wed, 2017-01-04 at 19:41 +0100, Rob Verduijn wrote:
> > > > > 2017-01-04 19:27 GMT+01:00 Dmitri Pal <dpal@redhat.com>:
> > > > >
> > > > > > On 01/04/2017 01:13 PM, Rob Verduijn wrote:
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2017-01-04 14:59 GMT+01:00 Simo Sorce <simo@redhat.com>:
> > > > > >
> > > > > >> On Wed, 2017-01-04 at 10:16 +0100, Rob Verduijn wrote:
> > > > > >> > ---------- Forwarded message ----------
> > > > > >> > From: Simo Sorce <simo@redhat.com>
> > > > > >> > Date: 2017-01-03 17:32 GMT+01:00
> > > > > >> > Subject: [gssproxy] Re: gssproxy broken on fedora
> > > > > >> > To: The GSS-Proxy developers and users mailing list <
> > > > > >> > gss-proxy@lists.fedorahosted.org>
> > > > > >> >
> > > > > >> >
> > > > > >> > On Mon, 2017-01-02 at 19:22 +0100, Rob Verduijn wrote:
> > > > > >> > >
> > > > > >> > > Nope that does not work on either fc24 or fc25.
> > > > > >> > > I did not try centos73 since it already worked on that
> > > > one.
> > > > > >> >
> > > > > >> > Given you tried manually, make sure you delete the ccache
> > > > before trying
> > > > > >> > with the client_keytab setting.
> > > > > >> >
> > > > > >> > If that doesn't work can you set debug = True in the
> > > > global section and
> > > > > >> > tell me if you get any useful output/error ?
> > > > > >> >
> > > > > >> > Simo.
> > > > > >> >
> > > > > >> > --
> > > > > >> > Simo Sorce * Red Hat, Inc * New York
> > > > > >> > _______________________________________________
> > > > > >> > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > > >> > To unsubscribe send an email to gss-proxy-leave@lists.fedo
> > > > rahosted.org
> > > > > >> >
> > > > > >> >
> > > > > >> > Hi,
> > > > > >> >
> > > > > >> > I checked for the cache, but there were no cache files
> > > > present in
> > > > > >> > /var/lib/gssproxy/clients.
> > > > > >> > I cleaned the sssd cache.
> > > > > >> > I set the debug entry, did a reboot, but also no log
> > > > entries appeared
> > > > > >> >
> > > > > >> > current /etc/gssproxy/gssproxy.conf
> > > > > >> >
> > > > > >> > [gssproxy]
> > > > > >> > debug=True
> > > > > >> >
> > > > > >> > [service/HTTP]
> > > > > >> >   mechs = krb5
> > > > > >> >   cred_store = keytab:/etc/gssproxy/http.keytab
> > > > > >> >   cred_store = ccache:/var/lib/gssproxy/clients/krb5cc_%U
> > > > > >> >   cred_store = client_keytab:/etc/gssproxy/http.keytab
> > > > > >> >   euid = 48
> > > > > >> >
> > > > > >> > and tested it with
> > > > > >> > su - apache -s /bin/bash
> > > > > >> >
> > > > > >> > The mount works fine for a regular ipa user on fedora
> > > > 24/25
> > > > > >> > according to systemctl status gssproxy the service is up
> > > > and running,
> > > > > >> >
> > > > > >> > [root@fedora-24 ~]# systemctl status gssproxy
> > > > > >> > ● gssproxy.service - GSSAPI Proxy Daemon
> > > > > >> >    Loaded: loaded
> > > > (/usr/lib/systemd/system/gssproxy.service; disabled;
> > > > > >> > vendor preset: disabled)
> > > > > >> >    Active: active (running) since Wed 2017-01-04 10:05:55
> > > > CET; 8min ago
> > > > > >> >  Main PID: 987 (gssproxy)
> > > > > >> >    CGroup: /system.slice/gssproxy.service
> > > > > >> >            └─987 /usr/sbin/gssproxy -D
> > > > > >> >
> > > > > >> > systemd[1]: Starting GSSAPI Proxy Daemon...
> > > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Debug Enabled
> > > > (level: 1)
> > > > > >> > gssproxy[972]: [2017/01/04 09:05:55]: Client connected (fd
> > > > =
> > > > > >> 10)[2017/01/04
> > > > > >> > 09:05:55]:  (pid = 987) (uid = 0) (gid = 0)[2017/01/04
> > > > 09:05:55]:
> > > > > >> (context
> > > > > >> > = system_u:system_r:kernel_t:s0)[2017/01/04 09:05:55]:
> > > > > >> > Started GSSAPI Proxy Daemon.
> > > > > >>
> > > > > >> If you turn on rpc.gssd debugging and kernel rpc debugging
> > > > do you see
> > > > > >> anything relevant ?
> > > > > >>
> > > > > >> Simo.
> > > > > >>
> > > > > >> --
> > > > > >> Simo Sorce * Red Hat, Inc * New York
> > > > > >> _______________________________________________
> > > > > >> gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > > >> To unsubscribe send an email to gss-proxy-leave@lists.fedora
> > > > hosted.org
> > > > > >>
> > > > > >
> > > > > >
> > > > > > It does not seem to look for the credits specified in the
> > > > gssproxy.conf
> > > > > > file.
> > > > > > How can I verify the running configuration of gssproxy ?
> > > > > >
> > > > > > Rob Verduijn
> > > > > >
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]:
> > > > #012handle_gssd_upcall:
> > > > > > 'mech=krb5 uid=48 enctypes=18,17,16,23,3,1,2 ' (nfs/clnt0)
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]:
> > > > krb5_not_machine_creds: uid 48
> > > > > > tgtname (null)
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: ERROR: GSS-API:
> > > > error in
> > > > > > gss_acquire_cred(): GSS_S_FAILURE (Unspecified GSS failure.
> > > > Minor code may
> > > > > > provide more information) - No Kerberos credentials available
> > > > (default
> > > > > > cache: KEYRING:persistent:48)
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client
> > > > creds with
> > > > > > uid 48 for server nfs.example.com in /tmp
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: CC
> > > > '/tmp/krb5ccmachine_EXAMPLE.COM'
> > > > > > being considered, with preferred realm 'EXAMPLE.COM'
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: CC
> > > > '/tmp/krb5ccmachine_EXAMPLE.COM'
> > > > > > owned by 0, not 48
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: looking for client
> > > > creds with
> > > > > > uid 48 for server nfs.example.com in /run/user/%U
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: Error doing scandir
> > > > on directory
> > > > > > '/run/user/48': No such file or directory
> > > > > > Jan  4 18:52:50 fedora-24 rpc.gssd[1034]: doing error
> > > > downcall
> > > > > >
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorah
> > > > osted.org
> > > > > >
> > > > > >
> > > > > > Why are you preferring credential cache in a file over a
> > > > keyring which is
> > > > > > default?
> > > > > > Have you tried without cred_store =
> > > > ccache:/var/lib/gssproxy/clients/krb5cc_%U
> > > > > > ?
> > > > > >
> > > > > > --
> > > > > > Thank you,
> > > > > > Dmitri Pal
> > > > > >
> > > > > > Engineering Director, Identity Management and Platform
> > > > Security
> > > > > > Red Hat, Inc.
> > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorah
> > > > osted.org
> > > > > >
> > > > > >
> > > > > Because it said so in the example here :
> > > > > https://fedorahosted.org/gss-proxy/wiki/Apache
> > > > >
> > > > > But  have tried it without and it still fails.
> > > > >
> > > > > Rob Verduijn
> > > > > _______________________________________________
> > > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahos
> > > > ted.org
> > > >
> > > > Rob is rpc.gssd running with the USE_GSS_PROXY=Yes environment
> > > > variable
> > > > on ?
> > > > If not then gssproxy is simply not involved here
> > > >
> > > > Simo.
> > > >
> > > > --
> > > > Simo Sorce * Red Hat, Inc * New York
> > > > _______________________________________________
> > > > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > > > To unsubscribe send an email to gss-proxy-leave@lists.fedorahoste
> > > > d.org
> > > >
> > >
> > >
> > > There is a
> > > GSS_USE_PROXY=yes in /etc/sysconfig/nfs
> > > so I added USE_GSS_PROXY=yes to it and also to the script
> > > /usr/libexec/nfs-utils/nfs-utils_env.sh
> > > so that it gets applied to the the file
> > > /run/sysconfig/nfs-utils at boot.
> > > I double checked after a reboot
> > > verified the share was working for an ordinary user
> > > but not for the apache user when using 'su - apache -s /bin/bash'
> > > also checked the kvno of the http.keytab just to make sure I wasn't
> > > missing the obvious.
> > > still no go.
> > >
> > > Rob Verduijn
> > >
> > >
> >
> > _______________________________________________
> > gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
> > To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.or
> > g

--
Simo Sorce * Red Hat, Inc. * New York
_______________________________________________
gss-proxy mailing list -- gss-proxy@lists.fedorahosted.org
To unsubscribe send an email to gss-proxy-leave@lists.fedorahosted.org