Alright, I think I've got everything* working. (* Not running the CA server on the Arm device, not tested, but from what I've read before I would need to adjust the startup timeout since OpenJDK is so slow).

1) I removed the Arm replica from the cluster and reimaged the device entirely and started over.

2) Rebuilt 389-ds-base using the patch by Mark Reynolds, and replica install succeeded (just as it did with the patch before). I did use the next version (1.3.8) of 389-ds-base because the package installed version bumped up to 1.3.8.1 - patch worked same as on 1.3.7

3) I also had to apply the fix to /etc/httpd/conf.d/ipa.conf from https://pagure.io/freeipa/issue/7337 so that Web UI login could work (I did this previously on the Arm device but still couldn't login - likely just too many things were tried and something else was busted elsewhere).

As far as I can tell everything seems to be working now.

I'm not sure if the patch provided would work as-is or if it would break on non-Arm architectures. 

The patch :

diff --git a/ldap/servers/plugins/replication/repl5_agmt.c b/ldap/servers/plugins/replication/repl5_agmt.c
index d71d3f38b..e0f1f41bd 100644
--- a/ldap/servers/plugins/replication/repl5_agmt.c
+++ b/ldap/servers/plugins/replication/repl5_agmt.c
@@ -3035,7 +3035,7 @@ agmt_update_maxcsn(Replica *r, Slapi_DN *sdn, int op, LDAPMod **mods, CSN *csn)
                 slapi_ch_free_string(&agmt->maxcsn);
                 agmt->maxcsn = slapi_ch_smprintf("%s;%s;%s;%ld;%d;%s", slapi_sdn_get_dn(agmt->replarea),
                                                  slapi_rdn_get_value_by_ref(slapi_rdn_get_rdn(agmt->rdn)), agmt->hostname,
-                                                 agmt->port, agmt->consumerRID, maxcsn);
+                                                 (long)agmt->port, (int)agmt->consumerRID, maxcsn);
             }
             PR_Unlock(agmt->lock);
         }

To get this fixed properly (to avoid building from source), should I open an issue in Pagure.io ? Or will someone else handle that?

On Wed, May 23, 2018 at 4:04 PM, Jonathan Vaughn <jonathan@creatuity.com> wrote:
Anyone have any further suggestions for troubleshooting this issue with the web UI?

On Fri, May 18, 2018 at 4:01 PM, Jonathan Vaughn <jonathan@creatuity.com> wrote:
httpd error_log

[Fri May 18 15:58:29.546255 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:29.547217 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI login_password.__call__:
[Fri May 18 15:58:29.549799 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Obtaining armor in ccache /var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:29.550812 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing anonymous ccache
[Fri May 18 15:58:29.552268 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:29.553377 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit -n -c /var/run/ipa/ccaches/armor_14004 -X X509_anchors=FILE:/var/kerberos/krb5kdc/kdc.crt -X X509_anchors=FILE:/var/lib/ipa-client/pki/kdc-ca-bundle.pem
[Fri May 18 15:58:30.486537 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.488137 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.489128 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.490955 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Initializing principal admin using password
[Fri May 18 15:58:30.492060 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using armor ccache /var/run/ipa/ccaches/armor_14004 for FAST webauth
[Fri May 18 15:58:30.493262 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Using enterprise principal
[Fri May 18 15:58:30.494728 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.495799 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kinit admin -c /var/run/ipa/ccaches/kinit_14004 -T /var/run/ipa/ccaches/armor_14004 -E
[Fri May 18 15:58:30.721637 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.723230 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=Password for admin@CREATUITY.INTERNAL:
[Fri May 18 15:58:30.723432 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284]
[Fri May 18 15:58:30.724345 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.725716 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Cleanup the armor ccache
[Fri May 18 15:58:30.727071 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting external process
[Fri May 18 15:58:30.728152 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: args=/usr/bin/kdestroy -A -c /var/run/ipa/ccaches/armor_14004
[Fri May 18 15:58:30.798642 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Process finished, return code=0
[Fri May 18 15:58:30.800249 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stdout=
[Fri May 18 15:58:30.801224 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: stderr=
[Fri May 18 15:58:30.857737 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTP connection (1): ipa-11.creatuity.internal
[Fri May 18 15:58:30.867978 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: http://ipa-11.creatuity.internal:80 "GET /ipa/session/cookie HTTP/1.1" 301 260
[Fri May 18 15:58:30.889415 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: Starting new HTTPS connection (1): ipa-11.creatuity.internal
[Fri May 18 15:58:31.105092 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: https://ipa-11.creatuity.internal:443 "GET /ipa/session/cookie HTTP/1.1" 200 0
[Fri May 18 15:58:31.157295 2018] [wsgi:error] [pid 14003:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.158347 2018] [wsgi:error] [pid 14003:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI jsonserver_session.__call__:
[Fri May 18 15:58:31.159289 2018] [wsgi:error] [pid 14003:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.160349 2018] [wsgi:error] [pid 14003:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login
[Fri May 18 15:58:31.175232 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI wsgi_dispatch.__call__:
[Fri May 18 15:58:31.176208 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: WSGI KerberosLogin.__call__:
[Fri May 18 15:58:31.177060 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: no ccache, need login
[Fri May 18 15:58:31.178047 2018] [wsgi:error] [pid 14004:tid 2903716672] [remote 10.10.11.3:59284] ipa: DEBUG: 401 Unauthorized need login


On Fri, May 18, 2018 at 3:53 PM, Rob Crittenden <rcritten@redhat.com> wrote:
Jonathan Vaughn wrote:
> System time appears to be fine.
>
> Here's the httpd access log (nothing in error log):
> 10.10.255.101 - - [18/May/2018:15:26:19 -0500] "GET /ipa/session/cookie
> HTTP/1.1" 301 260 "-" "python-requests/2.18.4"
> 10.10.255.101 - admin@CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500]
> "GET /ipa/session/cookie HTTP/1.1" 200 -
> 10.10.11.3 - - [18/May/2018:15:26:18 -0500] "POST
> /ipa/session/login_password HTTP/1.1" 200 20
> 10.10.11.3 - admin@CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "POST
> /ipa/session/json HTTP/1.1" 401 -
> 10.10.11.3 - admin@CREATUITY.INTERNAL [18/May/2018:15:26:20 -0500] "GET
> /ipa/session/login_kerberos?_=1526674892374 HTTP/1.1" 401 -
>

You'll want to look at the error log for more details. For even more
output create /etc/ipa/server.conf with the contents:

[global]
debug = True

and restart httpd

rob

>
> On Fri, May 18, 2018 at 3:23 PM, Jonathan Vaughn <jonathan@creatuity.com
> <mailto:jonathan@creatuity.com>> wrote:
>
>     Actually - may have spoke too soon. I thought I'd signed in to the
>     UI but I was actually in another server's UI. When I try to log in
>     it says that my login has expired... so progress but something else
>     is busted.
>
>     On Fri, May 18, 2018 at 3:20 PM, Jonathan Vaughn
>     <jonathan@creatuity.com <mailto:jonathan@creatuity.com>> wrote:
>
>         Looks like replica install finished just fine, and I can access
>         the the web UI on it. No obvious problems. 
>
>         So it would appear that code change fixed 389DS for ARM. 
>
>         I'm not running the Dogtag CA on the Pi, just on the non-Pi
>         servers, otherwise would need to probably adjust the startup
>         timeout for it so that the install script didn't give up waiting
>         on it. When I was first trying to play with this I tried
>         installing the first replica/master on the Pi, and ran into the
>         Dogtag timeout problem. Even installing Oracle Java instead of
>         the OpenJDK version of Java didn't solve it, because somehow it
>         always picks the OpenJDK java instead of oracle even if I change
>         the alternatives setting to default to the Oracle one... 
>
>         But as far as I can tell otherwise, everything is working as
>         expected.