Number of concurrent connections are decreased by replication.
by Jaehwan Kim
Hello.
I recently encoutered a poblem that nubmer of concurrent connecitons are decreased in FreeIPA servers.
[Architecutre - replication topology]
My replication topology which is circular (ring-shaped), consists of 13 FreeIPA servers.
These 13 servers are grouped as 3 clusters, of which members are 5, 4, 4 respectively.
NLBs(network load balancers) to share request from clients for ipa login, kerberos authenticaion, ldap connections, are assinged to each cluster.
Therefore 3 NLBs have 5, 4, 4 FreeIPA servers as their nlb backend pool, repectively.
This architecture has been worked successfully for 2 years, but recently I encountered a problem that 867 host_add per hours to one cluster results in "# of concurrent connections decrement" for all clusters.
Command to get # of concurrent connections is
dsconf -D "cn=Directory Manager" ldap://server.example.com monitor server | grep currentconnections:
About 2K connections are observed for each servers, by this command.
I also found that if servers which replication info isn't transfered to, this symptom doesn't happen, even though those are in the same replication topology ring.
Hence, I guess that "# of concurrent connections decrement" symptom is related to replcation.
I tried to tune the parameters like
dtablesize = 65535,
repl-release-timeout = 120,
nnsslapd-threadnumber = authomatic thread tuning,
db and entry cache auto-sizing (nsslapd-cache-autosize = 80,
with failure.
I want to ask help to solve this symptom, if posible.
Thank you.
JHK
2 months, 2 weeks
ipa-healthcheck timeout too short for ClonesConnectivyAndDataCheck
by Sam Morris
All three of my IPA servers have this health check failing:
[root@ipa3 ~]# ipa-healthcheck --source pki.server.healthcheck.clones.connectivity_and_data --check ClonesConnectivyAndDataCheck --output-type=human
Internal server error 'Link'
ERROR: pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck
[root@ipa5 ~]# ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=human
Internal server error 'Link'
Unable to reach KRA at https://ipa5.ipa.robots.org.uk:443: Request timed out
ERROR: pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck
ERROR: pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck
[root@ipa6 ~]# ipa-healthcheck --source=pki.server.healthcheck.clones.connectivity_and_data --check=ClonesConnectivyAndDataCheck --output-type=human
Unable to reach KRA at https://ipa6.ipa.robots.org.uk:443: Request timed out
ERROR: pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck
[root@ipa6 ~]# java.io.FileNotFoundException: /tmp/tmpc4h9ypzo/password.txt (No such file or directory)
at java.base/java.io.FileInputStream.open0(Native Method)
at java.base/java.io.FileInputStream.open(FileInputStream.java:216)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:157)
at java.base/java.io.FileInputStream.<init>(FileInputStream.java:111)
at java.base/java.io.FileReader.<init>(FileReader.java:60)
at com.netscape.cmstools.cli.MainCLI.loadPassword(MainCLI.java:241)
at com.netscape.cmstools.cli.MainCLI.parseOptions(MainCLI.java:416)
at com.netscape.cmstools.cli.MainCLI.execute(MainCLI.java:647)
at com.netscape.cmstools.cli.MainCLI.main(MainCLI.java:698)
The problem is the timeout mechanism implemented in
ipahealthcheck.core.core:run_plugin; it turns out that 10 seconds is not
sufficient in my case.
Plugin <pki.server.healthcheck.clones.connectivity_and_data.ClonesConnectivyAndDataCheck object at 0x7fe02c4992b0> elapsed time: 0:00:16.857277 sec
I've bumped the timeout to 30 seconds in
/etc/ipahealthcheck/ipahealthcheck.conf and all is well.
I wonder if others are running into this problem as well? If so it may
be worth making this change by default in ipa-healtcheck. Or maybe my
IPA servers are slower than everyone else's... :)
--
Sam Morris <https://robots.org.uk/>
PGP: rsa4096/CAAA AA1A CA69 A83A 892B 1855 D20B 4202 5CDA 27B9
2 months, 3 weeks
Automated enrollment of FreeIPA host
by Yuriy Halytskyy
We want to be able to destroy/recreate IPA enrolled hosts without
using admin credentials.
ipa-client-install with a keytab seems like a good option except it
generates a new keytab. And there is no non-hacky way of passing this
new keytab back to terraform. Can we tell it not to generate a new
keytab on re-enrollment?
Alternatively, we could create a user that has just enough permissions
to enroll host X but nothing else. What is the minimum set of
permissions for this?
Or is there a better way?
Cheers,
Yuriy
2 months, 3 weeks
Lots of named/dnssec related permission errors logged
by Harry G Coin
Hi!
What changes are necessary to avoid hundreds of log entries like this?
named[1511]: File.cpp(94): Could not open the file (Permission denied):
/var/lib/ipa/dnssec/tokens/95614bbc-437b-e1dc-3cd9-dac6fb2dafa2/abf57d53-0b7b-e8d2-d402-3b385005955d.object
Harry Coin
2 months, 3 weeks
DNS resolution failures
by Tania Hagan
Hi Freeipa-users,
We are currently running Freeipa version 4.9.11 on Rocky 8.8.
We have noticed over the last few months that external name resolution e.g. google.com fails to resolve on multiple Freeipa replicas even though the service named-pkcs11 remains up and running and journalctl or logs aren’t showing up any obvious errors to why this might be happening. We temporarily fix this by restarting the service, but the problem comes back at random times.
We currently have 39 DNS Zones
Our DNS Global Configuration has a forward policy of forward only, though the individual zones are set to forward first.
I’ve read a few articles that say maybe changing the forward policy might fix it, but nothing that mentions how to double check if changing the policy will fix it.
Are there any useful troubleshooting checks I could run to either help explain why our service keeps failing at random intervals or confirm any changes would fix the issue without the risk of potential downtime of our DNS service?
Many Thanks,
Tania
2 months, 3 weeks
Login failed due to an unknown reason
by Dan West
I am running into a strange issue with a few user accounts where logging into the web interface gives them the error message "Login failed due to an unknown reason”. It also prevents them from SSH’ing into IPA bound systems using passwords. Pubkeys work fine (as long as it is manually added to the local accounts) and any services I have bound to it (Gitlab, Mattermost, Owncloud, etc) seem to work fine. I ’think’ this is kerberos related since the only services that are using it is SSH and probably the IPA web interface. Here is the apache error log for it:
[Thu Jan 13 09:15:38.688228 2022] [wsgi:error] [pid 579266:tid 139812542121728] [remote xx.xxx.xx.xxx:52162] ipa: INFO: 401 Unauthorized: Major (851968): Unspecified GSS failure. Minor code may provide more information, Minor (2598844948): TGT has been revoked
I ’think’ the message "TGT has been revoked” is due to the 401 error, since the user is not showing as being authorized to login. However, this user is enabled and I have tried a number of things to try to fix it:
1. Disable/Re-enable account
2. Reset passwords
3. Kinit username (seems to get a ticket, but logins still do not work)
4. Run the account migration task (using the web gui)
5. Restart the IPA server and services
6. Re-initialize the IPA server from another master
Also, I can confirm that the passwords are correct since a failed password error message shows up differently and other services are using it correctly. Going down the Kerberos path, here is the krb5kdc log file:
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, WELLKNOWN/ANONYMOUS(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: NEEDED_PREAUTH: testuser(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM, Additional pre-authentication required
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): AS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: ISSUE: authtime 1642094138, etypes {rep=aes256-cts-hmac-sha1-96(18), tkt=aes256-cts-hmac-sha1-96(18), ses=aes256-cts-hmac-sha1-96(18)}, testuser(a)EXAMPLE.COM for krbtgt/EXAMPLE.COM(a)EXAMPLE.COM
Jan 13 09:15:38 ipa.example.com krb5kdc[579225](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser(a)EXAMPLE.COM for HTTP/ipa.example.com(a)EXAMPLE.COM, TGT has been revoked
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): closing down fd 12
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](Error): PAC issue: PAC record claims domain SID different to local domain SID or any trusted domain SID: local [S-1-5-21-997841278-3584560916-1456654135], PAC [S-1-5-21-2108153867-2082035330-3701898995]
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ : handle_authdata (-1765328364)
Jan 13 09:15:38 ipa.example.com krb5kdc[579226](info): TGS_REQ (7 etypes {aes256-cts-hmac-sha1-96(18), aes256-cts-hmac-sha384-192(20), camellia256-cts-cmac(26), aes128-cts-hmac-sha1-96(17), aes128-cts-hmac-sha256-128(19), camellia128-cts-cmac(25), DEPRECATED:arcfour-hmac(23)}) yy.yy.yy.yy: HANDLE_AUTHDATA: authtime 1642094138, etypes {rep=UNSUPPORTED:(0)} testuser(a)EXAMPLE.COM for HTTP/ipa.example.com(a)EXAMPLE.COM, TGT has been revoked
I only see two errors that might be related:
"PAC record claims domain SID different to local domain SID or any trusted domain SID”
"DEPRECATED:arcfour-hmac(23)”
However, those might just be red herrings or something else that is unrelated.
So far, there are only a small number of accounts that have this problem, but more seem to be popping up on a daily basis. The only fix I have found is the nuclear option, where I completely remove the account and then add it back in with the same UID/GID, group memberships and policies. After that it seems to work fine. However, I would rather not want to do this to all accounts since that would be a logistical nightmare.
Are there any suggestions for either troubleshooting or fixing this problem with a lighter approach? Is it possible to reset or regenerate the users kerberos authentication?
Thanks,
Dan West
Systems Administrator
Galois Inc.
http://galois.com
2 months, 3 weeks
=?utf-8?q?=5BFreeipa-users=5D?=(dnssec-keyfroml) of user ZZZ dumped core.
by Harry G Coin
FYI. El9, latest stable.
Mar 04 22:46:05 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipaserver.dnssec.bindmgr: INFO attrs: <ldap.cidict.cidict object at 0x7f283b53f7f0>
Mar 04 22:46:05 registry1.1.quietfountain.com named[1463]: client @0x7fd61803ae68 10.12.112.3#49016: received notify for zone '0.0.0.0.0.0.a.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:05 registry1.1.quietfountain.com named[1463]: client @0x7fd618191538 fc00:1002:c7::3#41560: received notify for zone '0.0.0.0.0.0.a.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(96): Missing log.level in configuration. Using default value: INFO
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(96): Missing slots.mechanisms in configuration. Using default value: ALL
Mar 04 22:46:05 registry1.1.quietfountain.com dnssec-keyfromlabel[75984]: Configuration.cpp(124): Missing slots.removable in configuration. Using default value: false
Mar 04 22:46:05 registry1.1.quietfountain.com systemd[1]: ipa-ods-exporter.service: Deactivated successfully.
Mar 04 22:46:05 registry1.1.quietfountain.com systemd[1]: ipa-ods-exporter.service: Consumed 4.047s CPU time.
Mar 04 22:46:05 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipaserver.dnssec.bindmgr: INFO attrs: <ldap.cidict.cidict object at 0x7f283b53faf0>
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(96): Missing log.level in configuration. Using default value: INFO
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(96): Missing slots.mechanisms in configuration. Using default value: ALL
Mar 04 22:46:06 registry1.1.quietfountain.com dnssec-keyfromlabel[75991]: Configuration.cpp(124): Missing slots.removable in configuration. Using default value: false
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: client @0x7fd61803ae68 10.12.112.3#49016: received notify for zone '0.0.0.0.0.0.f.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: client @0x7fd618191538 fc00:1002:c7::3#41560: received notify for zone '0.0.0.0.0.0.f.0.0.8.9.2.4.0.6.2.ip6.arpa'
Mar 04 22:46:06 registry1.1.quietfountain.com kernel: dnssec-keyfroml[75991]: segfault at 18 ip 00007f7fba586284 sp 00007ffff3e834a8 error 4 in libsofthsm2.so[7f7fba4ef000+9b000] likely on CPU 0 (core 0, socket 0)
Mar 04 22:46:06 registry1.1.quietfountain.com kernel: Code: 75 f4 48 8b 53 30 49 89 4c 24 20 49 89 44 24 10 49 89 54 24 28 48 83 c4 08 4c 89 e0 5b 41 5c c3 66 0f 1f 44 00 00 f3 0f 1e fa <48> 8b 47 18 48 85 c0 74 44 4c 8d 47 10 4c 89 c7 eb 12 66 2e 0f 1f
Mar 04 22:46:06 registry1.1.quietfountain.com systemd[1]: Started Process Core Dump (PID 75992/UID 0).
Mar 04 22:46:06 registry1.1.quietfountain.com systemd-coredump[75993]: Resource limits disable core dumping for process 75991 (dnssec-keyfroml).
Mar 04 22:46:06 registry1.1.quietfountain.com systemd-coredump[75993]: [🡕] Process 75991 (dnssec-keyfroml) of user 984 dumped core.
Mar 04 22:46:06 registry1.1.quietfountain.com systemd[1]:systemd-coredump@40-75992-0.service: Deactivated successfully.
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: Traceback (most recent call last):
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/libexec/ipa/ipa-dnskeysyncd", line 130, in <module>
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: while ldap_connection.syncrepl_poll(all=1, msgid=ldap_search):
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib64/python3.9/site-packages/ldap/syncrepl.py", line 464, in syncrepl_poll
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: self.syncrepl_refreshdone()
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/keysyncer.py", line 128, in syncrepl_refreshdone
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: self.bindmgr.sync(self.dnssec_zones)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 232, in sync
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: self.sync_zone(zone)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 205, in sync_zone
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: self.install_key(zone, uuid, attrs, tempdir)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib/python3.9/site-packages/ipaserver/dnssec/bindmgr.py", line 146, in install_key
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: result = ipautil.run(cmd, capture_output=True)
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: File "/usr/lib/python3.9/site-packages/ipapython/ipautil.py", line 599, in run
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: raise CalledProcessError(
Mar 04 22:46:07 registry1.1.quietfountain.com ipa-dnskeysyncd[75834]: ipapython.ipautil.CalledProcessError: CalledProcessError(Command ['/usr/sbin/dnssec-keyfromlabel', '-E', 'pkcs11', '-K', '/var/named/dyndb-ldap/ipa/master/_domainkey.rockstable.systems/XXXXXX', '-a', b'RSASHA256', '-l', b'pkcs11:object=YYYYY;pin-source=/var/lib/ipa/dnssec/softhsm_pin', '-P', b'20231101161909', '-A', 'none', '-I', 'none', '-D', 'none', '-E', 'pkcs11', '_domainkey.rockstable.systems.'] returned non-zero exit status -11: 'Found uninitialized token\nFound uninitialized token\nKey not found.\nPKCS11_load_public_key returned NULL\nFound uninitialized token\nKey not found.\nPKCS11_get_private_key returned NULL\ndnssec-keyfromlabel: warning: ENGINE_load_private_key failed (not found)\ndnssec-keyfromlabel: fatal: failed to get key _domainkey.rockstable.systems/RSASHA256: not found\n')
Mar 04 22:46:06 registry1.1.quietfountain.com named[1463]: dispatch 0x7fd618009200: shutting down due to TCP receive error: 2001:13c7:7002:3000::11#53: connection reset
2 months, 4 weeks
problem allowing Windows Active Directory users to access SMB shares on IPA client machine (IPA has trust with AD)
by Thomas Handler
Hi all,
I am facing a problem I got stuck upon.
We have the following setup:
+-----------+
| |
| AD |
| |
+-----------+
+--------------+ ^
| +----------+
| ums012 |
| |
| IPA |
+--------------+
^
| +--------------+
| | |
| | ums029 |
| | |
| | smbclient |
| +---+----------+
+------+--------+ |
| | |
| ums025 | |
| |<------------------+
| samba |
+---------------+
IPA has a trust established with AD which is working fine. Active Directory users can logon on Linux machines which are connected to IPA, `id some-ad-user` properly shows the AD groups.
ums012 and ums025 are running RHEL 9.3, ums029 is running RHEL 8.9.
ums029 is used as a test client via smbclient.
ums025 was setup following the instructions in https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/...
Setup worked fine, all steps went ok.
But when i switch over to ums029 and try to verify with an ad user I get
kinit <ad user>
smbclient -L ums025.idm.example.com -U <ad user> --use-kerberos=required
Password for [<ad user>@EXAMPLE.COM]:
gensec_spnego_client_negTokenInit_step: gse_krb5: creating NEG_TOKEN_INIT for cifs/ums025.idm.example.com failed (next[(null)]): NT_STATUS_INVALID_PARAMETER
session setup failed: NT_STATUS_INVALID_PARAMETER
whereas this is working fine when running the verification as IPA user.
I tried finding hints in the logs but was unsuccessful, thus I’m writing to the list.
Best regards,
Thomas
3 months