Hi FreeIPA list,

We are having an issue with our IPA environment of 4 replicated FreeIPA systems serving linux compute clients which join from a command in rc.local after boot. This worked in the past, but the system has been rebuilt since and the join command changed slightly. Unfortunately booting a few dozen nodes at a time, though they each talk to a different IPA system by design, leads to problems such as these - though 40-100 nodes can boot ok at a time there are always many stragglers, and the more we attempt to boot at once the more fail to join IPA (if we try to boot 500 nodes, we are lucky if we get a fifth of that joining IPA). Can you please advise on this output? Here is our join command in compute node rc.local:



ipa-client-install -U -q -p mach_join \
-w <redacted> \
--force-join \
--no-dns-sshfp \
--automount-location=redacted-node



And here is some log output of the 500 error:

ProtocolError: <ProtocolError for redacted.redacted.com/ipa/json: 500 Internal Server Error>
Cannot connect to the server due to generic error: cannot connect to 'https://redacted.redacted.com/ipa/json': Internal Server Error

As well as:

2021-06-02T21:39:11Z DEBUG Starting external process
2021-06-02T21:39:11Z DEBUG args=/usr/sbin/ipa-join -s redacted.redacted.com -b dc=redacted,dc=com -h redactednode.redacted.com -f
2021-06-02T21:40:13Z DEBUG Process finished, return code=17
2021-06-02T21:40:13Z DEBUG stdout=
2021-06-02T21:40:13Z DEBUG stderr=HTTP response code is 500, not 200
2021-06-02T21:40:13Z ERROR Joining realm failed: HTTP response code is 500, not 200 
And we also see timeouts happen:


2021-06-02T22:08:50Z DEBUG args=/usr/sbin/ipa-join -s redacted.redacted.com -b dc=redacted,dc=com -h redactednode.redacted.com -f
2021-06-02T22:09:01Z DEBUG Process finished, return code=17
2021-06-02T22:09:01Z DEBUG stdout=
2021-06-02T22:09:01Z DEBUG stderr=RPC failed at server. Configured time limit exceeded
2021-06-02T22:09:01Z ERROR Joining realm failed: RPC failed at server. Configured time limit exceeded
And we also see later timeouts near the end of the log in some cases though are able to authenticate and it didn't back out the install, but never got going healthy either:
2021-06-03T19:20:13Z DEBUG The ipa-client-install command failed, exception: TimeLimitExceeded: Configured time limit exceeded
Donald