Hi FreeIPA list,
We are having an issue with our IPA environment of 4 replicated FreeIPA
systems serving linux compute clients which join from a command in
rc.local after boot. This worked in the past, but the system has been
rebuilt since and the join command changed slightly. Unfortunately
booting a few dozen nodes at a time, though they each talk to a
different IPA system by design, leads to problems such as these - though
40-100 nodes can boot ok at a time there are always many stragglers, and
the more we attempt to boot at once the more fail to join IPA (if we try
to boot 500 nodes, we are lucky if we get a fifth of that joining IPA).
Can you please advise on this output? Here is our join command in
compute node rc.local:
ipa-client-install -U -q -p mach_join \
-w <redacted> \
--force-join \
--no-dns-sshfp \
--automount-location=redacted-node
And here is some log output of the 500 error:
ProtocolError: <ProtocolError for
redacted.redacted.com/ipa/json
<
http://redacted.redacted.com/ipa/json>: 500 Internal Server Error>
Cannot connect to the server due to generic error: cannot connect to
'https://redacted.redacted.com/ipa/json
<
https://hauth0003.dug.com/ipa/json>';: Internal Server Error
As well as:
2021-06-02T21:39:11Z DEBUG Starting external process
2021-06-02T21:39:11Z DEBUG args=/usr/sbin/ipa-join -s
redacted.redacted.com <
http://redacted.redacted.com> -b
dc=redacted,dc=com -h
redactednode.redacted.com
<
http://redactednode.redacted.com> -f
2021-06-02T21:40:13Z DEBUG Process finished, return code=17
2021-06-02T21:40:13Z DEBUG stdout=
2021-06-02T21:40:13Z DEBUG stderr=HTTP response code is 500, not 200
2021-06-02T21:40:13Z ERROR Joining realm failed: HTTP response code
is 500, not 200
And we also see timeouts happen:
2021-06-02T22:08:50Z DEBUG args=/usr/sbin/ipa-join -s
redacted.redacted.com <
http://redacted.redacted.com> -b
dc=redacted,dc=com -h
redactednode.redacted.com
<
http://redactednode.redacted.com> -f
2021-06-02T22:09:01Z DEBUG Process finished, return code=17
2021-06-02T22:09:01Z DEBUG stdout=
2021-06-02T22:09:01Z DEBUG stderr=RPC failed at server. Configured
time limit exceeded
2021-06-02T22:09:01Z ERROR Joining realm failed: RPC failed at
server. Configured time limit exceeded
And we also see later timeouts near the end of the log in some cases though are able to
authenticate and it didn't back out the install, but never got going healthy either:
2021-06-03T19:20:13Z DEBUG The ipa-client-install command failed,
exception: TimeLimitExceeded: Configured time limit exceeded
When you see Internal Error look to the Apache error log on the server
for more information.
In this case an LDAP search is failing because the server is too busy.
Look for queries failing with err=3 to get an idea of how long it is taking.
To increase the timeout use: ipa config-mod --searchtimelimit=INT
The default is 2 seconds.
You can pick a time at random but could see failures again.
rob