On 08/02/2017 12:11 AM, Florence Blanc-Renaud wrote:
On 08/02/2017 01:43 AM, Ian Harding wrote:
> On 08/01/2017 12:03 PM, Rob Crittenden wrote:
>> Ian Harding wrote:
>>> On 08/01/2017 07:39 AM, Florence Blanc-Renaud wrote:
>>>> On 08/01/2017 03:11 PM, Ian Harding wrote:
>>>>> On 08/01/2017 01:48 AM, Florence Blanc-Renaud wrote:
>>>>>> On 08/01/2017 01:32 AM, Ian Harding via FreeIPA-users wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 07/31/2017 11:34 AM, Rob Crittenden wrote:
>>>>>>>> Ian Harding via FreeIPA-users wrote:
>>>>>>>>> I had an unexpected restart of an IPA server that had
>>>>>>>>> apparently had
>>>>>>>>> updates run but had not been restarted. ipactl says
pki-tomcatd
>>>>>>>>> would
>>>>>>>>> not start.
>>>>>>>>>
>>>>>>>>> Strangely, the actual service appears to be running:
>>>>>>>>>
>>>>>>>>
>>>>>>>> dogtag is an application within tomcat so tomcat can run
without
>>>>>>>> dogtag
>>>>>>>> running.
>>>>>>>>
>>>>>>>> We need to see more of the dogtag debug log to see what
is
>>>>>>>> going on.
>>>>>>>>
>>>>>>>
>>>>>>> It looks like an authentication problem...
>>>>>>>
>>>>>>> [28/Jul/2017:10:08:47][localhost-startStop-1]: SSL handshake
>>>>>>> happened
>>>>>>> Could not connect to LDAP server host seattlenfs.bpt.rocks
port 636
>>>>>>> Error netscape.ldap.LDAPException: Authentication failed
(49)
>>>>>>>
>>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> dogtag stores its internal data in the LDAP server and needs to
>>>>>> establish a secure LDAP connection. You can check how this
>>>>>> connection is configured in /etc/pki/pki-tomcat/ca/CS.cfg, look
for
>>>>>> the lines:
>>>>>>
>>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>>> internaldb.ldapauth.bindDN=cn=Directory Manager
>>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
>>>>>> internaldb.ldapconn.host=vm-...
>>>>>> internaldb.ldapconn.port=636
>>>>>> internaldb.ldapconn.secureConn
>>>>>>
>>>>>> authtype can be SslClientAuth (authentication with a ssl
>>>>>> certificate) or BasicAuth (authentication with a bind DN and
>>>>>> password stored in /var/lib/pki/pki-tomcat/conf/password.conf).
>>>>>>
>>>>>> You can use this information to manually check the credentials.
For
>>>>>> instance with sslclientauth:
>>>>>>
>>>>>> export LDAPTLS_CACERTDIR=/etc/pki/pki-tomcat/alias
>>>>>> export LDAPTLS_CERT='subsystemCert cert-pki-ca'
>>>>>>
>>>>>> ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y
EXTERNAL
>>>>>> (provide the password from
/etc/pki/pki-tomcat/alias/pwdfile.txt)
>>>>>>
>>>>>
>>>>> I found this:
>>>>>
>>>>> internaldb.ldapauth.authtype=SslClientAuth
>>>>> internaldb.ldapauth.bindDN=uid=pkidbuser,ou=people,o=ipaca
>>>>> internaldb.ldapauth.bindPWPrompt=internaldb
>>>>> internaldb.ldapauth.clientCertNickname=subsystemCert cert-pki-ca
>>>>> internaldb.ldapconn.cloneReplicationPort=389
>>>>> ...
>>>>>
>>>>> and when I try the ldapsearch I am presented with a prompt to
provide
>>>>> a pin/password
>>>>>
>>>>> Please enter pin, password, or pass phrase for security token
>>>>> 'ldap(0)':
>>>>>
>>>>> but there is no password file...
>>>>>
>>>> Hi,
>>>>
>>>> you are right, in 4.4. there is no pwdfile.txt and the password can be
>>>> found in /var/lib/pki/pki-tomcat/conf/password.conf (with the tag
>>>> internal=...)
>>>>
>>>> Can you check if the password with the tag internal=... allows to read
>>>> the keys from the NSS db?
>>>> certutil -K -d /etc/pki/pki-tomcat/alias
>>>> (provide password)
>>>
>>> That works...
>>>
>>> # certutil -K -d /etc/pki/pki-tomcat/alias
>>> certutil: Checking token "NSS Certificate DB" in slot "NSS
User Private
>>> Key and Certificate Services"
>>> Enter Password or Pin for "NSS Certificate DB":
>>> < 0> rsa 0f327e760a7eecdcf6973f5dc57ca5367c592d64 (orphan)
>>> < 1> rsa b12580c7c696cfcd8aefc9405a7a870b24b7b96a NSS
>>> Certificate
>>> DB:auditSigningCert cert-pki-ca
>>> < 2> rsa 881b7254c40fa40bc50681bcc8d37bb3eb49937e caSigningCert
>>> cert-pki-ca
>>> < 3> rsa fa9a255a1d15585ac28064c0f4986e416bc48403 NSS
>>> Certificate
>>> DB:ocspSigningCert cert-pki-ca
>>> < 4> rsa 3fb609d0f7d72c2d325d6a2dc16577a7f7e5a01f Server-Cert
>>> cert-pki-ca
>>> < 5> rsa 1e9479a9556af9339bb5e4552ccbd381d3c38856 NSS
>>> Certificate
>>> DB:subsystemCert cert-pki-ca
>>>
>>> But this doesn't (with the same password from password.conf)
>>>
>>> # ldapsearch -H ldaps://`hostname`:636 -b "" -s base -Y EXTERNAL
>>> Please enter pin, password, or pass phrase for security token
>>> 'ldap(0)':
>>> SASL/EXTERNAL authentication started
>>> ldap_sasl_interactive_bind_s: Invalid credentials (49)
>>>
>>> That password is getting me somewhere though, since if I put in a
>>> nonsense or incorrect password it just prompts over and over.
>>
>> Let's step back a second. You upgraded from what to what?
>
> There wasn't much of a change... I just assumed someone ran yum
> upgrade and didn't restart, then the power outage... it looks like not
> much of a version change though.
>
> # grep ipa /var/log/yum.log
> Jan 08 04:45:32 Installed: ipa-common-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:45:32 Installed:
> ipa-client-common-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:46:06 Updated: libipa_hbac-1.14.0-43.el7_3.4.x86_64
> Jan 08 04:46:07 Updated: python-libipa_hbac-1.14.0-43.el7_3.4.x86_64
> Jan 08 04:46:08 Installed: python-ipaddress-1.0.16-2.el7.noarch
> Jan 08 04:47:04 Installed: python2-ipalib-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:05 Installed:
> python2-ipaclient-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:05 Updated: ipa-admintools-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:05 Installed:
> ipa-server-common-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:06 Installed:
> python2-ipaserver-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:07 Updated: sssd-ipa-1.14.0-43.el7_3.4.x86_64
> Jan 08 04:47:17 Updated: ipa-client-4.4.0-14.el7.centos.1.1.x86_64
> Jan 08 04:47:23 Updated: ipa-server-4.4.0-14.el7.centos.1.1.x86_64
> Jan 08 04:47:27 Updated: ipa-server-dns-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:47:36 Installed:
> ipa-python-compat-4.4.0-14.el7.centos.1.1.noarch
> Jan 08 04:50:07 Erased: ipa-python-4.2.0-15.0.1.el7.centos.19.x86_64
> Jul 28 09:55:41 Updated: ipa-common-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:42 Updated: ipa-client-common-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:43 Updated: ipa-server-common-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:43 Updated: python2-ipalib-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:44 Updated: python2-ipaclient-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:45 Updated: python2-ipaserver-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:45 Updated: ipa-admintools-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:46 Updated: ipa-client-4.4.0-14.el7.centos.7.x86_64
> Jul 28 09:55:48 Updated: ipa-server-4.4.0-14.el7.centos.7.x86_64
> Jul 28 09:55:48 Updated: ipa-server-dns-4.4.0-14.el7.centos.7.noarch
> Jul 28 09:55:48 Updated: ipa-python-compat-4.4.0-14.el7.centos.7.noarch
>
> # grep pki /var/log/yum.log
> Jan 08 04:46:05 Updated: pki-base-10.3.3-14.el7_3.noarch
> Jan 08 04:46:09 Updated: krb5-pkinit-1.14.1-27.el7_3.x86_64
> Jan 08 04:47:19 Installed: pki-base-java-10.3.3-14.el7_3.noarch
> Jan 08 04:47:19 Updated: pki-tools-10.3.3-14.el7_3.x86_64
> Jan 08 04:47:21 Updated: pki-server-10.3.3-14.el7_3.noarch
> Jan 08 04:47:22 Updated: pki-kra-10.3.3-14.el7_3.noarch
> Jan 08 04:47:22 Updated: pki-ca-10.3.3-14.el7_3.noarch
> Jul 28 10:13:16 Updated: pki-base-10.3.3-19.el7_3.noarch
> Jul 28 10:13:17 Updated: pki-base-java-10.3.3-19.el7_3.noarch
> Jul 28 10:13:17 Updated: pki-tools-10.3.3-19.el7_3.x86_64
> Jul 28 10:13:21 Updated: pki-server-10.3.3-19.el7_3.noarch
> Jul 28 10:13:21 Updated: pki-kra-10.3.3-19.el7_3.noarch
> Jul 28 10:13:21 Updated: pki-ca-10.3.3-19.el7_3.noarch
>
>
>>
>> Are you running in SELinux enforcing mode?
>
> # getenforce
> Enforcing
>
>>
>> If so can you run:
>>
>> # ausearch -m AVC
>>
>
> # ausearch -m AVC
> The NOLOG option to log_format is deprecated. Please use the
> write_logs option.
> The NOLOG option is overriding the write_logs current setting.
> ----
> time->Mon Mar 14 10:39:01 2016
> type=SYSCALL msg=audit(1457977141.767:17537): arch=c000003e syscall=2
> success=no exit=-13 a0=7fbcab038d10 a1=800 a2=1 a3=7fbca60f42e0
> items=0 ppid=1406 pid=12965 auid=4294967295 uid=0 gid=0 euid=581400001
> suid=0 fsuid=581400001 egid=581400001 sgid=0 fsgid=581400001
> tty=(none) ses=4294967295 comm="sshd" exe="/usr/sbin/sshd"
> subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 key=(null)
> type=AVC msg=audit(1457977141.767:17537): avc: denied { read } for
> pid=12965 comm="sshd" name="authorized_keys" dev="dm-2"
ino=116393517
> scontext=system_u:system_r:sshd_t:s0-s0:c0.c1023
> tcontext=system_u:object_r:unlabeled_t:s0 tclass=file
>
>
>> I suspect that the subsystem cert was renewed and everything wasn't
>> updated properly. If I'm right this is unrelated to the upgrade it just
>> shone a spotlight on it.
>>
>> Can you run these commands:
>>
>> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
>> uid=pkidbuser,ou=people,o=ipaca userCertificate description
>>
>
> This returns
>
> $ ldapsearch -LLL -D 'cn=directory manager' -W -b
> uid=pkidbuser,ou=people,o=ipaca userCertificate description
> Enter LDAP Password:
> dn: uid=pkidbuser,ou=people,o=ipaca
> userCertificate::
> MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
> ... stuff ...
> ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
> description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
> Subsystem,O=BPT.ROCKS
>
>> and
>>
>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca' | grep Serial
>> # certutil -L -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
>> cert-pki-ca' -a
>>
>
> These both return
>
> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
> cert-pki-ca' | grep Serial
> Enter Password or Pin for "NSS Certificate DB":
> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
> Unrecognized Object Identifier.
>
>
> # certutil -K -d /etc/pki/pki-tomcat/alias -n 'subsystemCert
> cert-pki-ca' -a
> certutil: Checking token "NSS Certificate DB" in slot "NSS User
> Private Key and Certificate Services"
> Enter Password or Pin for "NSS Certificate DB":
> certutil: problem listing keys: SEC_ERROR_UNRECOGNIZED_OID:
> Unrecognized Object Identifier.
>
Hi,
you made a typo and requested the keys (-K) instead of the certs (-L).
Please retry with -L and you should be able to see the certificate serial.
[root@seattlenfs ianh]# ldapsearch -LLL -D 'cn=directory manager' -W -b
uid=pkidbuser,ou=people,o=ipaca userCertificate description
Enter LDAP Password:
dn: uid=pkidbuser,ou=people,o=ipaca
userCertificate::
MIIDczCCAlugAwIBAgIBBDANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
ZNo4eX7jNT8Aa23gDl6xuLVa1SxlyE8XEo8dfAPXMJoFc=
description: 2;4;CN=Certificate Authority,O=BPT.ROCKS;CN=CA
Subsystem,O=BPT.ROCKS
So the serial is 4?
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' | grep Serial
Serial Number: 1341718541 (0x4ff9000d)
Which is NOT 4...
[root@seattlenfs ianh]# certutil -L -d /etc/pki/pki-tomcat/alias -n
'subsystemCert cert-pki-ca' -a
-----BEGIN CERTIFICATE-----
MIIDbjCCAlagAwIBAgIET/kADTANBgkqhkiG9w0BAQsFADA0MRIwEAYDVQQKDAlC
...
sjeiFbUFtoMnfmHLIYpe5QAR
-----END CERTIFICATE-----
And this cert is NOT the same.
Thank you again for your time!
Ian
Flo.
>
> which seems weird.
>
>> The description attribute in the ldapsearch output is of the form
>> 2;#;DN;DN
>>
>> The # should match the serial number in the first certutil output
>>
>> The ASCII blob (minus the BEGIN/END headers) should match one of the
>> userCertificate attributes.
>>
>> rob
>>
>
--
Ian Harding
IT Director
Brown Paper Tickets
1-800-838-3006 ext 7186
http://www.brownpapertickets.com