Hi FreeIPA list,
I don't see any in error log that match `
grep -i "err=3" /var/log/httpd/error_log`. We have tried raising searchtimelimit as high as 120, then 300 (now are trying 600) but observed no difference in the rate at which nodes succeeded or failed in IPA joins. We are somewhat puzzled by this, as none of the other values we are aware of might have changed, though it is possible that the IPA systems are under a little higher demand from client systems, we have tried to mitigate this by shutting down some workflows and aren't sure whether we've seen any improvement. Short of adjusting apache/resource/process timeouts it is difficult to say what might be wrong. To give an example, out of 250 nodes rebooted, only 112 joined IPA successfully. Here is some output from the error log, following what this looks like in the ipa client install log (error log output will match the node attempt):
2021-06-07T18:25:30Z DEBUG The ipa-client-install command failed, exception: NetworkError: cannot connect to 'https://redactednode.com/ipa/json': Internal Server Error
Different node, same time period:
[Mon Jun 07 13:24:02.178092 2021] [:error] [pid 25725] ipa: INFO: [xmlserver] mach_join@redacted.COM: join(u'redacted.node.com', nshardwareplatform=u'x86_64', nsosversion=u'3.10.0-1062.18.1.1.el7.redacted.x86_64', version=u'2.51'): TimeLimitExceeded
I also saw this:
[Mon Jun 07 13:25:07.103503 2021] [:error] [pid 25725] ipa: ERROR: non-public: IOError: request data read error
[Mon Jun 07 13:25:07.103529 2021] [:error] [pid 25725] Traceback (most recent call last):
[Mon Jun 07 13:25:07.103536 2021] [:error] [pid 25725] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in wsgi_execute
[Mon Jun 07 13:25:07.103542 2021] [:error] [pid 25725] data = read_input(environ)
[Mon Jun 07 13:25:07.103548 2021] [:error] [pid 25725] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in read_input
[Mon Jun 07 13:25:07.103553 2021] [:error] [pid 25725] return environ['wsgi.input'].read(length).decode('utf-8')
[Mon Jun 07 13:25:07.103559 2021] [:error] [pid 25725] IOError: request data read error
[Mon Jun 07 13:25:07.103826 2021] [:error] [pid 25725] ipa: INFO: [xmlserver] mach_join@redacted.COM: None: InternalError
[Mon Jun 07 13:25:07.149962 2021] [:error] [pid 25726] ipa: ERROR: non-public: IOError: request data read error
[Mon Jun 07 13:25:07.149984 2021] [:error] [pid 25726] Traceback (most recent call last):
[Mon Jun 07 13:25:07.149991 2021] [:error] [pid 25726] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 360, in wsgi_execute
[Mon Jun 07 13:25:07.149997 2021] [:error] [pid 25726] data = read_input(environ)
[Mon Jun 07 13:25:07.150002 2021] [:error] [pid 25726] File "/usr/lib/python2.7/site-packages/ipaserver/rpcserver.py", line 200, in read_input
[Mon Jun 07 13:25:07.150008 2021] [:error] [pid 25726] return environ['wsgi.input'].read(length).decode('utf-8')
[Mon Jun 07 13:25:07.150013 2021] [:error] [pid 25726] IOError: request data read error
After setting the timeout to 600 and rebooting the remaining 139 nodes from the initial set of 250, 83 joined of the 139 and we still had ISE occurring. In some cases, it would ISE on the first attempt, try another IPA system, and succeed. I'm not sure that even such a long timeout as 600 has helped.
Alfred