Hi,
- Are you using the targeted selinux policy? (what is the output of
"sestatus" command)
- are the selinux-policy / selinux-policy-targeted / ipa-selinux packages
up-to-date?
To troubleshoot further, I would first try to start named-pkcs11 in
permissive mode (setenforce 0; systemctl start named-pkcs11). If it works,
it means the error is related to SELinux. Go back in enforcing mode
(setenforce 1) and look for AVCs with
# date; systemctl start named-pkcs11
# ausearch -m AVC -ts recent
(look for AVCs happening after the date you started the service)
flo
On Mon, Aug 30, 2021 at 2:44 PM Jeremy Tourville <
jeremy_tourville(a)hotmail.com> wrote:
To answer your question, yes, /etc/named/ipa-ext.conf and
/etc/named/ipa-options-ext.conf exist.
When I attempted to start named*-pkcs11*.service. It failed. Journalctl
initially said there were issues with selinux. Anyhow, I attempted to
start the service again after making the selinux policy entries that were
suggested. I still was unable to get the service to start. Though, this
time I didn't get any selinux messages.
Here is what happened at the first start of named*-pkcs11*.service just
for reference:
[root@utility ~]# journalctl -xe
You can
generate a local policy module to allow this access.
Do
allow this
access for now by executing:
# ausearch
-c 'ipa-dnskeysync-' --raw | audit2allow -M my-ipadnskeysync
# semodule
-X 300 -i my-ipadnskeysync.pp
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]:
AnalyzeThread.run(): Set alarm timeout to 10
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]:
AnalyzeThread.run(): Cancel pending alarm
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]: SELinux
is preventing /usr/libexec/platform-python3.6 from 'read, write' acce>
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]: SELinux
is preventing /usr/libexec/platform-python3.6 from 'read, write' acce>
*****
Plugin catchall (100. confidence) suggests **************************
If you
believe that platform-python3.6 should be allowed read write access on>
Then you
should report this as a bug.
You can
generate a local policy module to allow this access.
Do
allow this
access for now by executing:
# ausearch
-c 'ipa-dnskeysync-' --raw | audit2allow -M my-ipadnskeysync
# semodule
-X 300 -i my-ipadnskeysync.pp
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]:
AnalyzeThread.run(): Set alarm timeout to 10
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]:
AnalyzeThread.run(): Cancel pending alarm
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]: SELinux
is preventing /usr/libexec/platform-python3.6 from lock access on the>
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]: SELinux
is preventing /usr/libexec/platform-python3.6 from lock access on the>
*****
Plugin catchall (100. confidence) suggests **************************
If you
believe that platform-python3.6 should be allowed lock access on the g>
Then you
should report this as a bug.
You can
generate a local policy module to allow this access.
Do
allow this
access for now by executing:
# ausearch
-c 'ipa-dnskeysync-' --raw | audit2allow -M my-ipadnskeysync
# semodule
-X 300 -i my-ipadnskeysync.pp
Aug 30 07:10:49
utility.idm.nac-issa.org setroubleshoot[21841]:
AnalyzeThread.run(): Set alarm timeout to 10
Here is the 2nd run after making the selinux entries.
[root@utility ~]# systemctl start named-pkcs11.service
Job for named-pkcs11.service failed because the control process exited
with error code.
See "systemctl status named-pkcs11.service" and "journalctl -xe" for
details.
[root@utility ~]# journalctl -xe
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: built with
'--build=x86_64-redhat-linux-gnu' '--host=x86_64-redhat-linux-gnu'
'>
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: running as:
named-pkcs11 -u named -c /etc/named.conf
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: compiled by
GCC 8.4.1 20200928 (Red Hat 8.4.1-1)
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: compiled
with libxml2 version: 2.9.7
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: linked to
libxml2 version: 20907
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: compiled
with libjson-c version: 0.13.1
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: linked to
libjson-c version: 0.13.1
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: compiled
with zlib version: 1.2.11
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: linked to
zlib version: 1.2.11
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: threads
support is enabled
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]:
----------------------------------------------------
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: BIND 9 is
maintained by Internet Systems Consortium,
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: Inc. (ISC),
a non-profit 501(c)(3) public-benefit
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]:
corporation. Support and training for BIND 9 are
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: available
at
https://www.isc.org/support
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]:
----------------------------------------------------
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: adjusted
limit on open files from 262144 to 1048576
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: found 4
CPUs, using 4 worker threads
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: using 3 UDP
listeners per interface
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: using up to
21000 sockets
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]:
initializing DST: PKCS#11 initialization failed
Aug 30 07:15:51
utility.idm.nac-issa.org named-pkcs11[22071]: exiting
(due to fatal error)
Aug 30 07:15:51
utility.idm.nac-issa.org systemd[1]:
named-pkcs11.service: Control process exited, code=exited status=1
Aug 30 07:15:51
utility.idm.nac-issa.org systemd[1]:
named-pkcs11.service: Failed with result 'exit-code'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support:
https://access.redhat.com/support
--
-- The unit named-pkcs11.service has entered the 'failed' state with
result 'exit-code'.
Aug 30 07:15:51
utility.idm.nac-issa.org systemd[1]: Failed to start
Berkeley Internet Name Domain (DNS) with native PKCS#11.
-- Subject: Unit named-pkcs11.service has failed
-- Defined-By: systemd
-- Support:
https://access.redhat.com/support
--
-- Unit named-pkcs11.service has failed.
--
-- The result is failed.
[root@utility ~]# cat /etc/named/ipa-ext.conf
// Custom managed file.
// Here you can set your own options, for instance ACL for recursion
access:
//
// acl "trusted_network" {
// localnets;
// localhost;
// 234.234.234.0/24;
// 2001::co:ffee:babe:1/48;
// };
// options {
// allow-recursion {trusted_network;};
// allow-query-cache {trusted_network;};
// };
//
// This file will NOT be overridden during updates!
[root@utility ~]# cat /etc/named/ipa-options-ext.conf
/* User customization for BIND named
*
* This file is included in /etc/named.conf and is not modified during IPA
* upgrades.
*
* It must only contain "options" settings. Any other setting must be
* configured in /etc/named/ipa-ext.conf.
*
* Examples:
* allow-recursion { trusted_network; };
* allow-query-cache { trusted_network; };
*/
/* turns on IPv6 for port 53, IPv4 is on by default for all ifaces */
listen-on-v6 { any; };
/* dnssec-enable is obsolete and 'yes' by default */
dnssec-validation yes;
[root@utility data]# systemctl status named-pkcs11.service
● named-pkcs11.service - Berkeley Internet Name Domain (DNS) with native
PKCS#11
Loaded: loaded (/usr/lib/systemd/system/named-pkcs11.service; disabled;
vendor preset: disabled)
Active: failed (Result: exit-code) since Mon 2021-08-30 07:27:50 CDT;
4min 49s ago
Process: 22249 ExecStart=/usr/sbin/named-pkcs11 -u named -c ${NAMEDCONF}
$OPTIONS (code=exited, status=1/FAILURE)
Process: 22244 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING"
== "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else
e>
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]:
----------------------------------------------------
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]: adjusted
limit on open files from 262144 to 1048576
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]: found 4
CPUs, using 4 worker threads
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]: using 3 UDP
listeners per interface
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]: using up to
21000 sockets
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]:
initializing DST: PKCS#11 initialization failed
Aug 30 07:27:50
utility.idm.nac-issa.org named-pkcs11[22250]: exiting
(due to fatal error)
Aug 30 07:27:50
utility.idm.nac-issa.org systemd[1]:
named-pkcs11.service: Control process exited, code=exited status=1
Aug 30 07:27:50
utility.idm.nac-issa.org systemd[1]:
named-pkcs11.service: Failed with result 'exit-code'.
Aug 30 07:27:50
utility.idm.nac-issa.org systemd[1]: Failed to start
Berkeley Internet Name Domain (DNS) with native PKCS#11.
[root@utility data]# journalctl -xe
Aug 30 07:27:53
utility.idm.nac-issa.org systemd[1]: Stopped PKI Tomcat
Server pki-tomcat.
-- Subject: Unit pki-tomcatd(a)pki-tomcat.service has finished shutting down
-- Defined-By: systemd
-- Support:
https://access.redhat.com/support
--
-- Unit pki-tomcatd(a)pki-tomcat.service has finished shutting down.
Aug 30 07:27:54
utility.idm.nac-issa.org ns-slapd[1665]:
[30/Aug/2021:07:27:54.054683013 -0500] - INFO - bdb_pre_close - Waiting for
4 databa>
Aug 30 07:27:55
utility.idm.nac-issa.org ns-slapd[1665]:
[30/Aug/2021:07:27:55.032053458 -0500] - INFO - bdb_pre_close - All
database threads>
Aug 30 07:27:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: ldap_sync_poll() failed
Aug 30 07:27:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
Aug 30 07:27:55
utility.idm.nac-issa.org ns-slapd[1665]:
[30/Aug/2021:07:27:55.054454093 -0500] - INFO -
ldbm_back_instance_set_destructor - >
Aug 30 07:27:55
utility.idm.nac-issa.org ns-slapd[1665]:
[30/Aug/2021:07:27:55.057417960 -0500] - INFO -
connection_post_shutdown_cleanup - s>
Aug 30 07:27:55
utility.idm.nac-issa.org ns-slapd[1665]:
[30/Aug/2021:07:27:55.059926010 -0500] - INFO - main - slapd stopped.
Aug 30 07:27:55
utility.idm.nac-issa.org systemd[1]:
dirsrv(a)IDM-NAC-ISSA-ORG.service: Succeeded.
-- Subject: Unit succeeded
-- Defined-By: systemd
-- Support:
https://access.redhat.com/support
--
-- The unit dirsrv(a)IDM-NAC-ISSA-ORG.service has successfully entered the
'dead' state.
Aug 30 07:27:55
utility.idm.nac-issa.org systemd[1]: Stopped 389
Directory Server IDM-NAC-ISSA-ORG..
-- Subject: Unit dirsrv(a)IDM-NAC-ISSA-ORG.service has finished shutting
down
-- Defined-By: systemd
-- Support:
https://access.redhat.com/support
--
-- Unit dirsrv(a)IDM-NAC-ISSA-ORG.service has finished shutting down.
Aug 30 07:27:59
utility.idm.nac-issa.org named[1527]: network unreachable
resolving 'a-ups-presencecore4-prod-azsc.eastus2.cloudapp.azure.com>
Aug 30 07:27:59
utility.idm.nac-issa.org named[1527]: network unreachable
resolving 'a-ups-presencecore4-prod-azsc.eastus2.cloudapp.azure.com>
Aug 30 07:28:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: bind to LDAP server failed
Aug 30 07:28:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
Aug 30 07:29:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: bind to LDAP server failed
Aug 30 07:29:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
Aug 30 07:30:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: bind to LDAP server failed
Aug 30 07:30:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
Aug 30 07:31:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: bind to LDAP server failed
Aug 30 07:31:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
Aug 30 07:32:55
utility.idm.nac-issa.org named[1527]: LDAP error: Can't
contact LDAP server: bind to LDAP server failed
Aug 30 07:32:55
utility.idm.nac-issa.org named[1527]: ldap_syncrepl will
reconnect in 60 seconds
It looks like I need to troubleshoot section 4 further..
auth_method, sasl_mech, sasl_user, all seem to be present in my /etc/named.conf
file
I was unable to find bind_dn, password, sasl_realm, sasl_password and
krb5_principal.
[root@utility data]# cat /etc/named.conf
options {
// turns on IPv6 for port 53, IPv4 is on by default for all ifaces
#listen-on-v6 {any;};
// Put files that named is allowed to write in the data/ directory:
directory "/var/named"; // the default
dump-file "data/cache_dump.db";
statistics-file "data/named_stats.txt";
memstatistics-file "data/named_mem_stats.txt";
// If not explicitly set, the ACLs for "allow-query-cache" and
// "allow-recursion" are set to "localnets; localhost;".
// If either "allow-query-cache" or "allow-recursion" is set,
// the other would be set the same value.
// Please refer to /etc/named/ipa-ext.conf
// for more information
tkey-gssapi-keytab "/etc/named.keytab";
pid-file "/run/named/named.pid";
dnssec-enable yes;
dnssec-validation yes;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
/* crypto policy snippet on platforms with system-wide policy. */
// not available
};
/* If you want to enable debugging, eg. using the 'rndc trace' command,
* By default, SELinux policy does not allow named to modify the
/var/named directory,
* so put the default debug log file in data/ :
*/
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
print-time yes;
};
};
zone "." IN {
type hint;
file "named.ca";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
/* custom configuration snippet */
include "/etc/named/ipa-ext.conf";
/* WARNING: This part of the config file is IPA-managed.
* Modifications may break IPA setup or upgrades.
*/
dyndb "ipa" "/usr/lib64/bind/ldap.so" {
uri "ldapi://%2fvar%2frun%2fslapd-IDM-NAC-ISSA-ORG.socket";
base "cn=dns,dc=idm,dc=nac-issa,dc=org";
server_id "utility.idm.nac-issa.org";
auth_method "sasl";
sasl_mech "GSSAPI";
sasl_user "DNS/utility.idm.nac-issa.org";
};
/* End of IPA-managed part. */
------------------------------
*From:* Florence Renaud <flo(a)redhat.com>
*Sent:* Monday, August 30, 2021 2:39 AM
*To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
*Cc:* Rob Crittenden <rcritten(a)redhat.com>; Jeremy Tourville <
jeremy_tourville(a)hotmail.com>
*Subject:* Re: [Freeipa-users] Re: Unable to start directory server after
updates
Hi,
on rhel8, IPA is using named*-pkcs11*.service, not named.service. In
order to manually start the bind service, you would need to use "systemctl
start named-pkcs11.service".
The journal may contain additional logs, as well as the output of
"systemctl status named-pkcs11.service".
IIRC in ipa 4.9, ipa introduced bind configuration snippets in
/etc/named/ipa-ext.conf and /etc/named/ipa-options-ext.conf. Do you have
such configuration files?
flo
On Sun, Aug 29, 2021 at 3:45 PM Jeremy Tourville via FreeIPA-users <
freeipa-users(a)lists.fedorahosted.org> wrote:
I found this page on troubleshooting -
https://docs.pagure.org/bind-dyndb-ldap/BIND9/NamedCannotStart.html
I can manually start named.service but cannot start named when using
ipactl.
*Section 1*
I was able to get a log (this log is prior to changes made in section 4)
#less /var/named/data/named.run
reloading configuration succeeded
reloading zones succeeded
network unreachable resolving './DNSKEY/IN': 2001:7fd::1#53
network unreachable resolving './DNSKEY/IN': 2001:500:1::53#53
network unreachable resolving './DNSKEY/IN': 2001:500:9f::42#53
network unreachable resolving './DNSKEY/IN': 2001:500:2f::f#53
network unreachable resolving './DNSKEY/IN': 2001:500:12::d0d#53
network unreachable resolving './DNSKEY/IN': 2001:500:200::b#53
network unreachable resolving './DNSKEY/IN': 2001:503:ba3e::2:30#53
network unreachable resolving './DNSKEY/IN': 2001:500:a8::e#53
network unreachable resolving './DNSKEY/IN': 2001:7fe::53#53
network unreachable resolving './DNSKEY/IN': 2001:dc3::35#53
network unreachable resolving './DNSKEY/IN': 2001:500:2::c#53
network unreachable resolving './DNSKEY/IN': 2001:500:2d::d#53
network unreachable resolving './DNSKEY/IN': 2001:503:c27::2:30#53
all zones loaded
running
managed-keys-zone: Key 20326 for zone . acceptance timer complete: key now
trusted
With the changes in section 4 (below) I now see this additional info in
the log:
received control channel command 'stop'
shutting down: flushing changes
stopping command channel on 127.0.0.1#953
stopping command channel on ::1#953
no longer listening on 127.0.0.1#53
no longer listening on ::1#53
exiting
I was unable to get a log from tmp/named_krb5.log using the rhel/fedora
method. Do I need to use the archlinux method?
*Section 2*
I don't see any evidence of this issue based on logs.
Furthermore, hostname FQDN and /etc/hosts are set properly according to
the examples shown
*Section 3*
The values here match
*Section 4*
I see that my system was running a named.conf file that didn't have any
credentials. I looked at my yum history and the timestamps for my
named.conf* files. The yum update that most likely affected them was run
at 9:52. The two oldest files are marked 9:55 and I presume are the
backups as part of the update process.
[root@utility etc]# ls -la named.conf*
-rw-r-----. 1 root named 1876 Aug 29 08:01 named.conf
-rw-r-----. 1 root named 1705 May 27 15:49 named.conf.bak
-rw-r--r--. 1 root root 1876 Aug 28 09:55 named.conf.ipa-backup
-rw-r-----. 1 root named 1535 Aug 28 09:55 named.conf.rpmsave
I did attempt to copy the oldest files over the existing named.conf and
start the named service. I still didn't have any luck in either case.
#cp named.conf.rpmsave named.conf
#ipactl start
#cp named.conf.ipa-backup named.conf
#ipactl start
Systemctl status when using named.conf.rpmsave version:
[root@utility etc]# systemctl status named
● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; linked; vendor
preset: disabled)
Active: active (running) since Sun 2021-08-29 08:38:05 CDT; 1s ago
Process: 2294 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF}
$OPTIONS (code=exited, status=0/SUCCESS)
Process: 2291 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING"
== "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else
ec>
Main PID: 2296 (named)
Tasks: 8 (limit: 37317)
Memory: 59.5M
CGroup: /system.slice/named.service
└─2296 /usr/sbin/named -u named -c /etc/named.conf
Aug 29 08:38:05
utility.idm.nac-issa.org named[2296]: managed-keys-zone:
Key 20326 for zone . acceptance timer complete: key now trusted
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: resolver priming
query complete
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: LDAP configuration
synchronization failed: socket is not connected
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: ldap_syncrepl will
reconnect in 60 seconds
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:f::1#53
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:c::1#53
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:40::1#53
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:48::1#53
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:b::1#53
Aug 29 08:38:06
utility.idm.nac-issa.org named[2296]: network unreachable
resolving '_ldap._tcp.idm.nac-issa.org/SRV/IN': 2001:500:e::1#53
Systemctl status when using named.conf.ipa-backup version:
[root@utility etc]# systemctl start named
[root@utility etc]# systemctl status named
● named.service - Berkeley Internet Name Domain (DNS)
Loaded: loaded (/usr/lib/systemd/system/named.service; linked; vendor
preset: disabled)
Active: active (running) since Sun 2021-08-29 08:33:54 CDT; 5s ago
Process: 2251 ExecStart=/usr/sbin/named -u named -c ${NAMEDCONF}
$OPTIONS (code=exited, status=0/SUCCESS)
Process: 2247 ExecStartPre=/bin/bash -c if [ ! "$DISABLE_ZONE_CHECKING"
== "yes" ]; then /usr/sbin/named-checkconf -z "$NAMEDCONF"; else
ec>
Main PID: 2252 (named)
Tasks: 8 (limit: 37317)
Memory: 64.7M
CGroup: /system.slice/named.service
└─2252 /usr/sbin/named -u named -c /etc/named.conf
Aug 29 08:33:55
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'eur2.akam.net/AAAA/IN': 2600:1401:1::43#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube2.idm.nac-issa.org/AAAA/IN': 2a00:edc0:107::1#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube2.idm.nac-issa.org/AAAA/IN': 2a00:edc0:107::49#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube2.idm.nac-issa.org/AAAA/IN': 2402:cf80:107::1#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube2.idm.nac-issa.org/AAAA/IN': 2402:cf80:107::49#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'nac-issa.org/DS/IN': 2001:500:c::1#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube1.idm.nac-issa.org/A/IN': 2402:cf80:107::1#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube1.idm.nac-issa.org/AAAA/IN': 2402:cf80:107::1#53
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube3.idm.nac-issa.org.idm.nac-issa.org/A/IN': 2402:cf80>
Aug 29 08:33:56
utility.idm.nac-issa.org named[2252]: network unreachable
resolving 'kube3.idm.nac-issa.org.idm.nac-issa.org/AAAA/IN': 2402:c>
Here are the contents of my file:
#less /etc/named.conf (named.conf.rpm version)
options {
// turns on IPv6 for port 53, IPv4 is on by default for all ifaces
listen-on-v6 {any;};
// Put files that named is allowed to write in the data/ directory:
directory "/var/named"; // the default
dump-file "data/cache_dump.db";
statistics-file "data/named_stats.txt";
memstatistics-file "data/named_mem_stats.txt";
// If not explicitly set, the ACLs for "allow-query-cache" and
// "allow-recursion" are set to "localnets; localhost;".
// If either "allow-query-cache" or "allow-recursion" is
set,
// the other would be set the same value.
// Please refer to /etc/named/ipa-ext.conf
// for more informations
tkey-gssapi-keytab "/etc/named.keytab";
pid-file "/run/named/named.pid";
dnssec-enable yes;
dnssec-validation yes;
/* Path to ISC DLV key */
bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
/* crypto policy snippet on platforms with system-wide policy. */
// not available
};
/* If you want to enable debugging, eg. using the 'rndc trace' command,
* By default, SELinux policy does not allow named to modify the
/var/named directory,
* so put the default debug log file in data/ :
*/
logging {
channel default_debug {
file "data/named.run";
severity dynamic;
print-time yes;
};
};
zone "." IN {
type hint;
file "named.ca";
};
include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";
/* custom configuration snippet */
include "/etc/named/ipa-ext.conf";
/* WARNING: This part of the config file is IPA-managed.
* Modifications may break IPA setup or upgrades.
*/
dyndb "ipa" "/usr/lib64/bind/ldap.so" {
uri "ldapi://%2fvar%2frun%2fslapd-IDM-NAC-ISSA-ORG.socket";
base "cn=dns, dc=idm,dc=nac-issa,dc=org";
server_id "utility.idm.nac-issa.org";
auth_method "sasl";
sasl_mech "GSSAPI";
sasl_user "DNS/utility.idm.nac-issa.org";
};
/* End of IPA-managed part. */
I also compared the two oldest files but I am not sure what changes should
be made in my existing named.conf.
# diff named.conf.rpmsave named.conf.ipa-backup
1,9d0
< /* WARNING: This config file is managed by IPA.
< *
< * DO NOT MODIFY! Any modification will be overwritten by upgrades.
< *
< *
< * - /etc/named/ipa-options-ext.conf (for options)
< * - /etc/named/ipa-ext.conf (all other settings)
< */
<
10a2,4
> // turns on IPv6 for port 53, IPv4 is on by default for all ifaces
> listen-on-v6 {any;};
>
17c11,16
< tkey-gssapi-keytab "/etc/named.keytab";
---
> // If not explicitly set, the ACLs for "allow-query-cache" and
> // "allow-recursion" are set to "localnets; localhost;".
> // If either "allow-query-cache" or "allow-recursion" is set,
> // the other would be set the same value.
> // Please refer to /etc/named/ipa-ext.conf
> // for more informations
18a18
> tkey-gssapi-keytab "/etc/named.keytab";
21c21,25
< managed-keys-directory "/var/named/dynamic";
---
> dnssec-enable yes;
> dnssec-validation yes;
>
> /* Path to ISC DLV key */
> bindkeys-file "/etc/named.iscdlv.key";
23,24c27
< /* user customizations of options */
< include "/etc/named/ipa-options-ext.conf";
---
> managed-keys-directory "/var/named/dynamic";
50c53
< /* user customization */
---
> /* custom configuration snippet */
52a56,58
> /* WARNING: This part of the config file is IPA-managed.
> * Modifications may break IPA setup or upgrades.
> */
55c61
< base "cn=dns,dc=idm,dc=nac-issa,dc=org";
---
> base "cn=dns, dc=idm,dc=nac-issa,dc=org";
60a67
> /* End of IPA-managed part. */
------------------------------
*From:* Jeremy Tourville <jeremy_tourville(a)hotmail.com>
*Sent:* Saturday, August 28, 2021 7:07 PM
*To:* freeipa-users(a)lists.fedorahosted.org <
freeipa-users(a)lists.fedorahosted.org>
*Cc:* Rob Crittenden <rcritten(a)redhat.com>
*Subject:* Re: [Freeipa-users] Unable to start directory server after
updates
OK, I quickly realized I couldn't yum/dnf downgrade as I still had a
version/data mismatch. Now I understand what the error means. I did the
latter part of my previous question and performed an ipa-server-upgrade.
....
.....
The IPA services were upgraded
The ipa-server-upgrade command was successful
Now I tried to start my ipa server but had limited success. Named service
won't start
....
....
Starting named Service
Failed to start named Service
Shutting down
I tried to force and see what else would have issues
#ipactl start --ignore-service-failure
....
....
Failed to start named Service
Forced start, ignoring named Service, continuing normal operation
....
....
Starting ipa-dnskeysyncd Service
Failed to start ipa-dnskeysyncd Service
Forced start, ignoring ipa-dnskeysyncd Service, continuing normal operation
ipa: INFO: The ipactl command was successful
Here is the entire sequence-
[root@utility slapd-IDM-NAC-ISSA-ORG]# ipa-server-upgrade
Upgrading IPA:. Estimated time: 1 minute 30 seconds
[1/9]: saving configuration
[2/9]: disabling listeners
[3/9]: enabling DS global lock
[4/9]: disabling Schema Compat
[5/9]: starting directory server
[6/9]: updating schema
[7/9]: upgrading server
[8/9]: stopping directory server
[9/9]: restoring configuration
Done.
Update complete
Upgrading IPA services
Upgrading the configuration of the IPA services
Disabled p11-kit-proxy
[Verifying that root certificate is published]
[Migrate CRL publish directory]
CRL tree already moved
[Verifying that KDC configuration is using ipa-kdb backend]
[Fix DS schema file syntax]
Syntax already fixed
[Removing RA cert from DS NSS database]
RA cert already removed
[Enable sidgen and extdom plugins by default]
[Updating HTTPD service IPA configuration]
[Updating HTTPD service IPA WSGI configuration]
Nothing to do for configure_httpd_wsgi_conf
[Migrating from mod_nss to mod_ssl]
Already migrated to mod_ssl
[Moving HTTPD service keytab to gssproxy]
[Removing self-signed CA]
[Removing Dogtag 9 CA]
[Checking for deprecated KDC configuration files]
[Checking for deprecated backups of Samba configuration files]
[Remove FILE: prefix from 'dedicated keytab file' in Samba configuration]
[Update 'max smbd processes' in Samba configuration to prevent unlimited
SMBLoris attack amplification]
dnssec-validation yes
[Add missing CA DNS records]
IPA CA DNS records already processed
DNS service is not configured
[Upgrading CA schema]
CA schema update complete
[Update certmonger certificate renewal configuration]
Certmonger certificate renewal configuration already up-to-date
[Enable PKIX certificate path discovery and validation]
PKIX already enabled
[Authorizing RA Agent to modify profiles]
[Authorizing RA Agent to manage lightweight CAs]
[Ensuring Lightweight CAs container exists in Dogtag database]
[Adding default OCSP URI configuration]
[Disabling cert publishing]
pki-tomcat configuration changed, restart pki-tomcat
[Ensuring CA is using LDAPProfileSubsystem]
[Migrating certificate profiles to LDAP]
Migrating profile 'caECServerCertWithSCT'
Migrating profile 'caServerCertWithSCT'
Migrating profile 'caServerKeygen_DirUserCert'
Migrating profile 'caServerKeygen_UserCert'
[Ensuring presence of included profiles]
[Add default CA ACL]
Default CA ACL already added
[Updating ACME configuration]
[Migrating to authselect profile]
Already migrated to authselect profile
[Create systemd-user hbac service and rule]
hbac service systemd-user already exists
[Add root(a)IDM.NAC-ISSA.ORG alias to admin account]
Alias already exists
[Setup SPAKE]
[Setup PKINIT]
[Enable server krb5.conf snippet]
[Adding ipa-ca alias to HTTP certificate]
Resubmitting HTTP cert tracking request
The IPA services were upgraded
The ipa-server-upgrade command was successful
[root@utility slapd-IDM-NAC-ISSA-ORG]# ipactl start
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Failed to start named Service
Shutting down
Hint: You can use --ignore-service-failure option for forced start in case
that a non-critical service failed
Aborting ipactl
[root@utility slapd-IDM-NAC-ISSA-ORG]# ipactl start
--ignore-service-failure
Existing service file detected!
Assuming stale, cleaning and proceeding
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Failed to start named Service
Forced start, ignoring named Service, continuing normal operation
Starting httpd Service
Starting ipa-custodia Service
Starting pki-tomcatd Service
Starting smb Service
Starting winbind Service
Starting ipa-otpd Service
Starting ipa-dnskeysyncd Service
Failed to start ipa-dnskeysyncd Service
Forced start, ignoring ipa-dnskeysyncd Service, continuing normal operation
ipa: INFO: The ipactl command was successful
[root@utility slapd-IDM-NAC-ISSA-ORG]#
------------------------------
*From:* Jeremy Tourville <jeremy_tourville(a)hotmail.com>
*Sent:* Saturday, August 28, 2021 6:45 PM
*To:* freeipa-users(a)lists.fedorahosted.org <
freeipa-users(a)lists.fedorahosted.org>
*Cc:* Rob Crittenden <rcritten(a)redhat.com>
*Subject:* Re: [Freeipa-users] Unable to start directory server after
updates
CentOS Linux release 8.4.2105
VERSION: 4.9.2, API_VERSION: 2.240
Prior to any updates I was at ver 8.2 of CentOS
The shared library was loaded and now I can start dirsrv. THANKS! That's
definitely big a step in the right direction. As I thought, my upgrade
looks like it caused the version be too new for the existing dirsrv data.
I thought I had set my OS distro release version and that is my own dumb
mistake...
IPA version error: data needs to be upgraded (expected version
'4.9.2-4.module_el8.4.0+846+96522ed7', current version
'4.8.4-7.module_el8.2.0+374+0d2d74a1')
I am thinking I could downgrade to get things up and running or do you
suggest upgrading the data to work with the application version I have
installed?
[root@utility slapd-IDM-NAC-ISSA-ORG]# ipactl status
Directory Service: RUNNING
krb5kdc Service: STOPPED
kadmin Service: STOPPED
named Service: STOPPED
httpd Service: STOPPED
ipa-custodia Service: STOPPED
pki-tomcatd Service: STOPPED
smb Service: STOPPED
winbind Service: STOPPED
ipa-otpd Service: STOPPED
ipa-dnskeysyncd Service: STOPPED
9 service(s) are not running
[root@utility slapd-IDM-NAC-ISSA-ORG]# ipactl start
IPA version error: data needs to be upgraded (expected version
'4.9.2-4.module_el8.4.0+846+96522ed7', current version
'4.8.4-7.module_el8.2.0+374+0d2d74a1')
Automatically running upgrade, for details see /var/log/ipaupgrade.log
Be patient, this may take a few minutes.
Starting Directory Service
Starting krb5kdc Service
Starting kadmin Service
Starting named Service
Failed to start named Service
Shutting down
Hint: You can use --ignore-service-failure option for forced start in case
that a non-critical service failed
Aborting ipactl
------------------------------
*From:* Rob Crittenden <rcritten(a)redhat.com>
*Sent:* Saturday, August 28, 2021 5:31 PM
*To:* FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
*Cc:* Jeremy Tourville <jeremy_tourville(a)hotmail.com>
*Subject:* Re: [Freeipa-users] Unable to start directory server after
updates
Jeremy Tourville via FreeIPA-users wrote:
> I was doing some maintenance and updates this morning. At some point I
noticed I couldn't reach the web interface anymore. My server has been up
and running for the last year and is not a new install. I reviewed
//var/log/dirsrv/slapd-IDM-NAC-ISSA-ORG/errors. I also confirmed I did not
have disk space issues.
>
> Here is part of my log file:
> [28/Aug/2021:10:46:35.380380540 -0500] - INFO - slapd_daemon - slapd
started. Listening on All Interfaces port 389 for LDAP requests
> [28/Aug/2021:10:46:35.383040751 -0500] - INFO - slapd_daemon - Listening
on All Interfaces port 636 for LDAPS requests
> [28/Aug/2021:10:46:35.385415998 -0500] - INFO - slapd_daemon - Listening
on /var/run/slapd-IDM-NAC-ISSA-ORG.socket for LDAPI requests
> [28/Aug/2021:10:46:35.439358079 -0500] - ERR - schema-compat-plugin -
schema-compat-plugin tree scan will start in about 5 seconds!
> [28/Aug/2021:10:46:40.494600578 -0500] - WARN - str2entry_dupcheck -
Duplicate value for attribute type memberUid detected in entry
cn=sudo-infra,cn=groups,cn=compat,dc=idm,dc=nac-issa,dc=org. Extra value
ignored.
> [28/Aug/2021:10:46:40.527665958 -0500] - WARN - str2entry_dupcheck -
Duplicate value for attribute type memberUid detected in entry
cn=sudo-devel,cn=groups,cn=compat,dc=idm,dc=nac-issa,dc=org. Extra value
ignored.
> [28/Aug/2021:10:46:40.560185359 -0500] - ERR - schema-compat-plugin -
warning: no entries set up under cn=computers,
cn=compat,dc=idm,dc=nac-issa,dc=org
> [28/Aug/2021:10:46:40.582782578 -0500] - ERR - schema-compat-plugin -
Finished plugin initialization.
> [28/Aug/2021:11:20:49.697931599 -0500] - INFO - op_thread_cleanup -
slapd shutting down - signaling operation threads - op stack size 4 max
work q size 2 max work q stack size 2
> [28/Aug/2021:11:20:49.706989092 -0500] - INFO - slapd_daemon - slapd
shutting down - closing down internal subsystems and plugins
> [28/Aug/2021:11:20:49.724450159 -0500] - INFO - bdb_pre_close - Waiting
for 4 database threads to stop
> [28/Aug/2021:11:20:51.131059518 -0500] - INFO - bdb_pre_close - All
database threads now stopped
> [28/Aug/2021:11:20:51.152587508 -0500] - INFO -
ldbm_back_instance_set_destructor - Set of instances destroyed
> [28/Aug/2021:11:20:51.155514615 -0500] - INFO -
connection_post_shutdown_cleanup - slapd shutting down - freed 2 work q
stack objects - freed 7 op stack objects
> [28/Aug/2021:11:20:51.158002944 -0500] - INFO - main - slapd stopped.
> [28/Aug/2021:13:14:20.585994349 -0500] - NOTICE - config_set_port -
Non-Secure Port Disabled
> [28/Aug/2021:13:14:20.607117053 -0500] - ERR - symload_report_error -
Netscape Portable Runtime error -5977:
/usr/lib64/dirsrv/plugins/libipa_cldap.so: cannot open shared object file:
No such file or directory
> [28/Aug/2021:13:14:20.609768545 -0500] - ERR - symload_report_error -
Could not open library "/usr/lib64/dirsrv/plugins/libipa_cldap.so" for
plugin ipa_cldap
> [28/Aug/2021:13:14:20.612257544 -0500] - ERR - load_plugin_entry -
Unable to load plugin "cn=ipa_cldap,cn=plugins,cn=config"
> [28/Aug/2021:13:14:21.012890173 -0500] - ERR - symload_report_error -
Netscape Portable Runtime error -5977:
/usr/lib64/dirsrv/plugins/libipa_cldap.so: cannot open shared object file:
No such file or directory
> [28/Aug/2021:13:14:21.018097465 -0500] - ERR - symload_report_error -
Could not open library "/usr/lib64/dirsrv/plugins/libipa_cldap.so" for
plugin ipa_cldap
> [28/Aug/2021:13:14:21.020655816 -0500] - ERR - load_plugin_entry -
Unable to load plugin "cn=ipa_cldap,cn=plugins,cn=config"
> [28/Aug/2021:13:15:53.219524942 -0500] - ERR - symload_report_error -
Netscape Portable Runtime error -5977:
/usr/lib64/dirsrv/plugins/libipa_cldap.so: cannot open shared object file:
No such file or directory
> [28/Aug/2021:13:15:53.228547473 -0500] - ERR - symload_report_error -
Could not open library "/usr/lib64/dirsrv/plugins/libipa_cldap.so" for
plugin ipa_cldap
> [28/Aug/2021:13:15:53.231054342 -0500] - ERR - load_plugin_entry -
Unable to load plugin "cn=ipa_cldap,cn=plugins,cn=config"
> [28/Aug/2021:13:17:13.917125368 -0500] - NOTICE - config_set_port -
Non-Secure Port Disabled
> [28/Aug/2021:13:17:13.932712979 -0500] - ERR - symload_report_error -
Netscape Portable Runtime error -5977:
/usr/lib64/dirsrv/plugins/libipa_cldap.so: cannot open shared object file:
No such file or directory
> [28/Aug/2021:13:17:13.935253118 -0500] - ERR - symload_report_error -
Could not open library "/usr/lib64/dirsrv/plugins/libipa_cldap.so" for
plugin ipa_cldap
> [28/Aug/2021:13:17:13.937761206 -0500] - ERR - load_plugin_entry -
Unable to load plugin "cn=ipa_cldap,cn=plugins,cn=config"
>
> Can anyone offer troubleshooting suggestions? Do you need a debug file
or is this log enough? Thanks in advance for your input!
Knowing the distribution and version would help.
This missing shared library is provided by [free]ipa-server-trust-ad,
ipa-server, or something like it depending on the release.
rob
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure