Hi Fraser,

I got one FreeIPA server that use the original jss in F31. It had problem in upgrading.
But it resume upgrading after upgrading to a newer jss. However it still have java/tomcat security problem described in https://bugzilla.redhat.com/show_bug.cgi?id=1755634

I had another FreeIPA server (a master replica of the above server) that could not finish upgrading with the updated jss in early November.

Since I have very few data in the above FreeIPA domain, I replaced it by creating a new one with CentOS.

Thanks,
Patrick


On Wed, Nov 27, 2019 at 6:45 PM Fraser Tweedale <ftweedal@redhat.com> wrote:
Hi Patrick,

I want to follow up with this.  Did you get things working again?

With the latest packages for both f30 and f31, I upgraded a FreeIPA
installation from f30 to f31 without encountering any problems.

Perhaps the jss issue caused the system to enter a poor state during
the initial ipa-server-upgrade, such that subsequent runs of
ipa-server-upgrade fail, even after upgrading jss.

Anyone else have thoughts on this?  Anyone able to reproduce with
latest packages?

Thanks,
Fraser

On Tue, Nov 05, 2019 at 03:25:13AM +0800, Patrick Dung via
FreeIPA-users wrote:
> The stack trace of my last mail should be hitting this bug:
> https://bugzilla.redhat.com/show_bug.cgi?id=1755634
>
> On Fri, Nov 1, 2019 at 3:20 PM Patrick Dung <patdung100@gmail.com>
> wrote:
>
> > Hello all, 1) Resent, I used reply instead of reply all in the
> > last mail.
> >
> > 2) I tailed the dirsrv log and perform a manual
> > ipa-server-upgrade.  I didn't found connection refused log in
> > the dirsrv.
> >
> > 3) BTW, I had another ipa server that is a replica. Originally
> > both freeipa server had upgrade problem.  On the replica server,
> > I tried to install jss-4.6.2-2.fc31.x86_64 (according to
> > https://bugzilla.redhat.com/show_bug.cgi?id=1766451) The
> > ipa-server-upgrade is successfully run on the replica server.
> > But there is problem when I access: https://replica:8443, the
> > error message is shown in below.  HTTP Status 500 – Internal
> > Server Error ------------------------------
> >
> > *Type* Exception Report
> >
> > *Message* org.apache.jasper.JasperException: Unable to compile
> > class for JSP
> >
> > *Description* The server encountered an unexpected condition
> > that prevented it from fulfilling the request.
> >
> > *Exception*
> >
> > org.apache.jasper.JasperException:
> > org.apache.jasper.JasperException: Unable to compile class for
> > JSP
> > org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:604)
> > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:422)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
> > java.security.AccessController.doPrivileged(Native Method)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
> >
> > *Root Cause*
> >
> > org.apache.jasper.JasperException: Unable to compile class for
> > JSP
> > org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:619)
> > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
> > java.security.AccessController.doPrivileged(Native Method)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
> >
> > *Root Cause*
> >
> > java.security.AccessControlException: access denied
> > ("java.util.PropertyPermission"
> > "tolerateIllegalAmbiguousVarargsInvocation" "read")
> > java.security.AccessControlContext.checkPermission(AccessControlContext.java:472)
> > java.security.AccessController.checkPermission(AccessController.java:886)
> > java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
> > java.lang.SecurityManager.checkPropertyAccess(SecurityManager.java:1294)
> > java.lang.System.getProperty(System.java:717)
> > org.eclipse.jdt.internal.compiler.impl.CompilerOptions.<init>(CompilerOptions.java:513)
> > org.apache.jasper.compiler.JDTCompiler.generateClass(JDTCompiler.java:483)
> > org.apache.jasper.compiler.Compiler.compile(Compiler.java:392)
> > org.apache.jasper.compiler.Compiler.compile(Compiler.java:362)
> > org.apache.jasper.compiler.Compiler.compile(Compiler.java:346)
> > org.apache.jasper.JspCompilationContext.compile(JspCompilationContext.java:603)
> > org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:399)
> > org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:385)
> > org.apache.jasper.servlet.JspServlet.service(JspServlet.java:329)
> > javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
> > java.security.AccessController.doPrivileged(Native Method)
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
> > sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > java.lang.reflect.Method.invoke(Method.java:498)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
> > org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
> > java.security.AccessController.doPrivileged(Native Method)
> > javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
> > org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
> > org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
> >
> > *Note* The full stack trace of the root cause is available in
> > the server logs.
> >
> >
> > 4) On the main server, I tried to upgrade jss and run
> > ipa-server-upgrade but the error is still there. The error is
> > the same as the original mail a few days ago.
> >
> > Thanks,
> >
> > Patrick
> >
> > On Thu, Oct 31, 2019 at 12:25 PM Patrick Dung
> > <patdung100@gmail.com> wrote:
> >
> >> I tail the dirsrv log and perform a manual ipa-server-upgrade.
> >> I didn't found connection refused log in the dirsrv.
> >>
> >> Thanks, Patrick
> >>
> >> On Thu, Oct 31, 2019 at 7:43 AM Fraser Tweedale
> >> <ftweedal@redhat.com> wrote:
> >>
> >>> Is there anything in the dirsrv log relating to the connection
> >>> attempt?  Connection Refused could in fact be a TLS handshake
> >>> error (the TLS handshake also includes certificate
> >>> authentication).
> >>>
> >>> Cheers, Fraser
> >>>
> >>> On Wed, Oct 30, 2019 at 10:47:54PM +0800, Patrick Dung via
> >>> FreeIPA-users wrote:
> >>> > Hello Endi,
> >>> >
> >>> > The DS is up.
> >>> >
> >>> > $ ldapsearch -LLL -x -H ldaps://home.local.nonet:636 -b
> >>> > "cn=users,cn=accounts,dc=local,dc=nonet" dn:
> >>> > cn=users,cn=accounts,dc=local,dc=nonet objectClass: top
> >>> > objectClass: nsContainer cn: users
> >>> >
> >>> > dn: uid=admin,cn=users,cn=accounts,dc=local,dc=nonet
> >>> > objectClass: top objectClass: person objectClass:
> >>> > posixaccount objectClass: krbprincipalaux objectClass:
> >>> > krbticketpolicyaux objectClass: inetuser objectClass:
> >>> > ipaobject objectClass: ipasshuser objectClass:
> >>> > ipaSshGroupOfPubKeys uid: admin cn: Administrator sn:
> >>> > Administrator uidNumber: 700000 gidNumber: 700000
> >>> > homeDirectory: /home/admin loginShell: /bin/bash gecos:
> >>> > Administrator
> >>> >
> >>> > I had include more logs as attachment.
> >>> >
> >>> > Thanks, Patrick
> >>> >
> >>> > On Wed, Oct 30, 2019 at 10:23 PM Endi Sukma Dewata
> >>> > <edewata@redhat.com
> >>> >
> >>> > wrote:
> >>> >
> >>> > > Hi Patrick,
> >>> > >
> >>> > > I see that you included the CA debug log:
> >>> > >
> >>> > > 2019-10-30 05:03:50 [main] FINE: LdapBoundConnection:
> >>> > > Connecting to home.local.nonet:636 with client cert auth
> >>> > > 2019-10-30 05:03:50 [main] FINE:
> >>> ldapconn/PKISocketFactory.makeSSLSocket:
> >>> > > begins 2019-10-30 05:03:50 [main] SEVERE: Unable to create
> >>> > > socket: java.net.ConnectException: Connection refused
> >>> > > (Connection refused) java.net.ConnectException: Connection
> >>> > > refused (Connection refused) at
> >>> > > java.net.PlainSocketImpl.socketConnect(Native Method) at
> >>> > > java.net
> >>> > > .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> >>> > > at java.net
> >>> > >
> >>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> >>> > >         at java.net
> >>> > >         .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> >>> > >         at
> >>> > >         java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>> > >         at java.net.Socket.connect(Socket.java:607) at
> >>> > >         java.net.Socket.connect(Socket.java:556) at
> >>> > >         java.net.Socket.<init>(Socket.java:452) at
> >>> > >         java.net.Socket.<init>(Socket.java:262)
> >>> > >
> >>> > > The stack trace above is incomplete so it's hard to tell
> >>> > > exactly
> >>> where
> >>> > > in PKI code the exception happened, but the earlier
> >>> > > message seem to indicate that it's trying to connect to
> >>> > > the the DS, so you need to
> >>> make
> >>> > > sure the DS is running and accessible.
> >>> > >
> >>> > > -- Endi S. Dewata
> >>> > >
> >>> > > ----- Original Message -----
> >>> > > > Hi Patrick,
> >>> > > >
> >>> > > > The "javax.ws.rs.ServiceUnavailableException: Subsystem
> >>> unavailable"
> >>> > > > suggests that the CA subsystem got undeployed
> >>> > > > automatically from Tomcat due to an error in CA (the
> >>> > > > Tomcat itself seems to be running just fine).
> >>> > > >
> >>> > > > You'll need to check the CA debug log in
> >>> /var/log/pki/pki-tomcat/ca/*,
> >>> > > > hopefully it will show the actual problem. Once it's
> >>> > > > fixed, you
> >>> just
> >>> > > > need to restart Tomcat and the CA should be redeployed
> >>> automatically.
> >>> > > >
> >>> > > > Also check the upgrade log in
> >>> > > > /var/log/pki/pki-server-upgrade-* to
> >>> see
> >>> > > > if there's any PKI upgrade issue.
> >>> > > >
> >>> > > > Hope this helps.
> >>> > > >
> >>> > > > -- Endi S. Dewata
> >>> > > >
> >>> > > > ----- Original Message -----
> >>> > > > > Looks like it's the second problem (on pagure)
> >>> > > > >
> >>> > > > > -- Logs begin at Wed 2019-10-30 02:34:10 HKT, end at
> >>> > > > > Wed
> >>> 2019-10-30
> >>> > > > > 06:28:21 HKT. -- Oct 30 03:39:43 home.local.nonet
> >>> > > > > systemd[1]: Starting PKI Tomcat
> >>> Server
> >>> > > > > pki-tomcat...  Oct 30 03:39:44 home.local.nonet
> >>> > > > > pki-server[57211]: ---------------------------- Oct 30
> >>> > > > > 03:39:44 home.local.nonet pki-server[57211]:
> >>> > > > > pki-tomcat
> >>> instance
> >>> > > > > migrated Oct 30 03:39:44 home.local.nonet
> >>> > > > > pki-server[57211]: ---------------------------- Oct 30
> >>> > > > > 03:39:44 home.local.nonet systemd[1]: Started PKI
> >>> > > > > Tomcat
> >>> Server
> >>> > > > > pki-tomcat.  Oct 30 03:39:44 home.local.nonet
> >>> > > > > server[57330]: Java virtual
> >>> machine
> >>> > > used:
> >>> > > > > /usr/lib/jvm/jre-1.8.0-openjdk/bin/java Oct 30
> >>> > > > > 03:39:44 home.local.nonet server[57330]: classpath
> >>> > > > > used:
> >>> > > > >
> >>> > >
> >>> /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/lib/java/commons-daemon.jar
> >>> > > > > Oct 30 03:39:44 home.local.nonet server[57330]: main
> >>> > > > > class used: org.apache.catalina.startup.Bootstrap Oct
> >>> > > > > 30 03:39:44 home.local.nonet server[57330]: flags
> >>> > > > > used: -Djava.library.path=/usr/lib64/nuxwdog-jni Oct
> >>> > > > > 30 03:39:44 home.local.nonet server[57330]: options
> >>> > > > > used: -Dcatalina.base=/var/lib/pki/pki-tomcat
> >>> > > -Dcatalina.home=/usr/share/tomcat
> >>> > > > > -Djava.endorsed.dirs=
> >>> -Djava.io.tmpdir=/var/lib/pki/pki-tomcat/temp
> >>> > > > >
> >>> > >
> >>> -Djava.util.logging.config.file=/var/lib/pki/pki-tomcat/conf/logging.properties
> >>> > > > > -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
> >>> > > > > -Djava.security.manager
> >>> > > > >
> >>> -Djava.security.policy==/var/lib/pki/pki-tomcat/conf/catalina.policy
> >>> > > > > Oct 30 03:39:44 home.local.nonet server[57330]:
> >>> > > > > arguments used:
> >>> start
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]:
> >>> > > > > WARNING:
> >>> Exception
> >>> > > > > processing realm
> >>> > > > > [com.netscape.cms.tomcat.ProxyRealm@296c31c9]
> >>> > > > > background
> >>> process
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]:
> >>> > > > > javax.ws.rs.ServiceUnavailableException: Subsystem
> >>> > > > > unavailable Oct 30 03:40:03 home.local.nonet
> >>> > > > > server[57330]: at
> >>> > > > >
> >>> > >
> >>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:142)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1137)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5566)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > > java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > >
> >>> > >
> >>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >>> > > > > Oct 30 03:40:03 home.local.nonet server[57330]: at
> >>> > > > > java.lang.Thread.run(Thread.java:748)
> >>> > > > >
> >>> > > > > The pki-tomcat instance is running. It would output
> >>> > > > > error when I
> >>> browse
> >>> > > > > https://home.local.nonet:8443/ca , but
> >>> > > https://home.local.nonet:8443/pki/
> >>> > > > > is ok Instance ID: pki-tomcat Active: True Unsecure
> >>> > > > > Port: 8080 Secure Port: 8443 AJP Port: 8009 Tomcat
> >>> > > > > Port: 8005
> >>> > > > >
> >>> > > > > CA Subsystem: Type: Subordinate CA (Security Domain)
> >>> > > > > SD Registration URL: https://home.local.nonet:443
> >>> > > > > Enabled: True Unsecure URL:
> >>> > > > > http://home.local.nonet:8080/ca/ee/ca Secure Agent
> >>> > > > > URL: https://home.local.nonet:8443/ca/agent/ca Secure
> >>> > > > > EE URL: https://home.local.nonet:8443/ca/ee/ca Secure
> >>> > > > > Admin URL: https://home.local.nonet:8443/ca/services
> >>> > > > > PKI Console URL: https://home.local.nonet:8443/ca
> >>> > > > >
> >>> > > > > Thanks, Patrick
> >>> > > > >
> >>> > > > > On Wed, Oct 30, 2019 at 5:44 AM Alex Scheel <
> >>> > > > > ascheel@redhat.com
> >>> >
> >>> > > wrote:
> >>> > > > >
> >>> > > > >
> >>> > > > > You might try checking journalctl output.
> >>> > > > >
> >>> > > > > It might be this bug:
> >>> > > https://bugzilla.redhat.com/show_bug.cgi?id=1766451
> >>> > > > >
> >>> > > > > Otherwise, it is a perfect match for this bug:
> >>> > > > > https://pagure.io/dogtagpki/issue/3111
> >>> > > > >
> >>> > > > > Which I'd also like journalctl output on, if you have
> >>> > > > > any to
> >>> share. :)
> >>> > > > >
> >>> > > > >
> >>> > > > > I should have a Bodhi update out tonight yet for the
> >>> > > > > issue in
> >>> the BZ.
> >>> > > > > Without more information, I'm not sure we'd know what
> >>> > > > > cause for the
> >>> second
> >>> > > issue
> >>> > > > > is.
> >>> > > > >
> >>> > > > > - Alex
> >>> > > > >
> >>> > > > > ----- Original Message -----
> >>> > > > > > From: "Patrick Dung via FreeIPA-users" <
> >>> > > > > > freeipa-users@lists.fedorahosted.org > To:
> >>> > > > > > freeipa-users@lists.fedorahosted.org Cc: "Patrick
> >>> > > > > > Dung" < patdung100@gmail.com > Sent: Tuesday,
> >>> > > > > > October 29, 2019 5:29:09 PM Subject: [Freeipa-users]
> >>> > > > > > FreeIPA having problem after
> >>> upgrading from
> >>> > > > > > Fedora 30 to 31
> >>> > > > > >
> >>> > > > > > Hello,
> >>> > > > > >
> >>> > > > > > I got problem upgrading from FC30 to FC31.  Before
> >>> > > > > > upgrade the FreeIPA in FC30 is running fine.
> >>> > > > > >
> >>> > > > > > After OS upgrade, IPA cannot start and checked that
> >>> > > > > > it stuck
> >>> at the
> >>> > > CA
> >>> > > > > > part.  I run ipa-server-upgrade manually but there
> >>> > > > > > is problem.
> >>> > > > > >
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG request GET
> >>> > > > > > https://home.local.nonet:8443/ca/rest/account/login
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG request body ''
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG response status 500
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG response headers
> >>> > > > > > Content-Type: text/html;charset=utf-8
> >>> > > > > > Content-Language: en Content-Length: 2481 Date: Tue,
> >>> > > > > > 29 Oct 2019 21:03:58 GMT Connection: close
> >>> > > > > >
> >>> > > > > >
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG response body (decoded):
> >>> > > > > > b'<!doctype html><html lang="en"><head><title>HTTP
> >>> > > > > > Status 500 \xe2\x80\x93 Internal
> >>> Server
> >>> > > > > > Error</title><style type="text/css">h1
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
> >>> > > > > > h2
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:16px;}
> >>> > > > > > h3
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:14px;}
> >>> > > > > > body
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;}
> >>> > > > > > b
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;}
> >>> > > > > > p
> >>> > > > > >
> >>> > >
> >>> {font-family:Tahoma,Arial,sans-serif;background:white;color:black;font-size:12px;}
> >>> > > > > > a {color:black;} a.name {color:black;} .line
> >>> > > > > >
> >>> > >
> >>> {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP
> >>> > > > > > Status 500 \xe2\x80\x93 Internal Server
> >>> > > > > > Error</h1><hr
> >>> class="line"
> >>> > > > > > /><p><b>Type</b> Exception
> >>> > > > > > Report</p><p><b>Message</b>
> >>> Subsystem
> >>> > > > > > unavailable</p><p><b>Description</b> The server
> >>> > > > > > encountered an
> >>> > > unexpected
> >>> > > > > > condition that prevented it from fulfilling the
> >>> > > > > > request.</p><p><b>Exception</b></p><pre> javax.ws.rs
> >>> > > > > > .ServiceUnavailableException: Subsystem
> >>> > > > > >
> >>> > >
> >>> unavailable\n\tcom.netscape.cms.tomcat.ProxyRealm.findSecurityConstraints(ProxyRealm.java:150)\n\torg.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:515)\n\tcom.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:82)\n\torg.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)\n\torg.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)\n\torg.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)\n\torg.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)\n\torg.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)\n\torg.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)\n\
> >>> > > > > > torg.apache.tomcat.util.net
> >>> > > > > > .NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1589)\n\
> >>> > > > > > torg.apache.tomcat.util.net
> >>> > > > > >
> >>> > >
> >>> .SocketProcessorBase.run(SocketProcessorBase.java:49)\n\tjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\torg.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)\n\tjava.lang.Thread.run(Thread.java:748)\n</pre><p><b>Note</b>
> >>> > > > > > The full stack trace of the root cause is available
> >>> > > > > > in the
> >>> server
> >>> > > > > > logs.</p><hr class="line" /><h3>Apache
> >>> > > Tomcat/9.0.26</h3></body></html>'
> >>> > > > > > 2019-10-29T21:03:58Z ERROR IPA server upgrade
> >>> > > > > > failed: Inspect /var/log/ipaupgrade.log and run
> >>> > > > > > command ipa-server-upgrade
> >>> manually.
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG File
> >>> > > > > > "/usr/lib/python3.7/site-packages/ipapython/admintool.py",
> >>> line 179,
> >>> > > in
> >>> > > > > > execute return_value = self.run() File
> >>> > > > > >
> >>> > >
> >>> "/usr/lib/python3.7/site-packages/ipaserver/install/ipa_server_upgrade.py",
> >>> > > > > > line 54, in run server.upgrade() File
> >>> > > > > >
> >>> > >
> >>> "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py",
> >>> > > > > > line 2223, in upgrade upgrade_configuration() File
> >>> > > > > >
> >>> > >
> >>> "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py",
> >>> > > > > > line 2093, in upgrade_configuration
> >>> > > > > > ca_enable_ldap_profile_subsystem(ca) File
> >>> > > > > >
> >>> > >
> >>> "/usr/lib/python3.7/site-packages/ipaserver/install/server/upgrade.py",
> >>> > > > > > line 414, in ca_enable_ldap_profile_subsystem
> >>> > > > > > cainstance.migrate_profiles_to_ldap() File
> >>> > > "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py",
> >>> > > > > > line 1937, in migrate_profiles_to_ldap
> >>> > > > > > _create_dogtag_profile(profile_id, profile_data,
> >>> overwrite=False)
> >>> > > > > > File
> >>> > > "/usr/lib/python3.7/site-packages/ipaserver/install/cainstance.py",
> >>> > > > > > line 1943, in _create_dogtag_profile with
> >>> > > > > > api.Backend.ra_certprofile as profile_api: File
> >>> "/usr/lib/python3.7/site-packages/ipaserver/plugins/dogtag.py",
> >>> > > line
> >>> > > > > > 1315, in __enter__ raise
> >>> > > > > > errors.RemoteRetrieveError(reason=_('Failed to
> >>> authenticate to
> >>> > > CA
> >>> > > > > > REST API'))
> >>> > > > > >
> >>> > > > > > 2019-10-29T21:03:58Z DEBUG The ipa-server-upgrade
> >>> > > > > > command
> >>> failed,
> >>> > > > > > exception: RemoteRetrieveError: Failed to
> >>> > > > > > authenticate to CA
> >>> REST API
> >>> > > > > > 2019-10-29T21:03:58Z ERROR Unexpected error - see
> >>> > > /var/log/ipaupgrade.log
> >>> > > > > > for details: RemoteRetrieveError: Failed to
> >>> > > > > > authenticate to CA REST API 2019-10-29T21:03:58Z
> >>> > > > > > ERROR The ipa-server-upgrade command
> >>> failed. See
> >>> > > > > > /var/log/ipaupgrade.log for more information
> >>> > > > > >
> >>> > > > > > From /var/log/pki/pki-tomcat/ca/debug log file:
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE: LdapAuthInfo:
> >>> > > > > > init() 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapAuthInfo: init begins 2019-10-30 05:03:50 [main]
> >>> > > > > > FINEST: Getting
> >>> > > > > > internaldb.ldapauth.authtype=SslClientAuth
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE: LdapAuthInfo: init
> >>> > > > > > ends 2019-10-30 05:03:50 [main] FINEST: Property
> >>> internaldb.errorIfDown
> >>> > > not
> >>> > > > > > found 2019-10-30 05:03:50 [main] FINEST: Getting
> >>> > > internaldb.errorIfDown=true
> >>> > > > > > 2019-10-30 05:03:50 [main] FINEST: Property
> >>> internaldb.doCloning not
> >>> > > > > > found 2019-10-30 05:03:50 [main] FINEST: Getting
> >>> internaldb.doCloning=true
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory:
> >>> doCloning:
> >>> > > true
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory:
> >>> mininum: 3
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory:
> >>> maximum: 15
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory: host: home.local.nonet
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory: port:
> >>> 636
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory: secure:
> >>> true
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory:
> >>> > > authentication: 2
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > LdapBoundConnFactory: makeConnection(true)
> >>> > > > > > 2019-10-30 05:03:50 [main] FINEST: Getting
> >>> > > > > > internaldb.ldapauth.clientCertNickname=subsystemCert
> >>> cert-pki-ca
> >>> > > > > > 2019-10-30 05:03:50 [main] FINEST: Property
> >>> > > > > > tcp.keepAlive not
> >>> found
> >>> > > > > > 2019-10-30 05:03:50 [main] FINEST: Getting
> >>> > > > > > tcp.keepAlive=true 2019-10-30 05:03:50 [main] FINE:
> >>> > > > > > TCP Keep-Alive: true 2019-10-30 05:03:50 [main]
> >>> > > > > > FINE: LdapBoundConnection:
> >>> Connecting to
> >>> > > > > > home.local.nonet:636 with client cert auth
> >>> > > > > > 2019-10-30 05:03:50 [main] FINE:
> >>> > > ldapconn/PKISocketFactory.makeSSLSocket:
> >>> > > > > > begins 2019-10-30 05:03:50 [main] SEVERE: Unable to
> >>> > > > > > create socket: java.net.ConnectException: Connection
> >>> > > > > > refused (Connection
> >>> refused)
> >>> > > > > > java.net.ConnectException: Connection refused
> >>> > > > > > (Connection
> >>> refused)
> >>> > > > > > at java.net.PlainSocketImpl.socketConnect(Native
> >>> > > > > > Method) at java.net
> >>> > > > > >
> >>> .AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> >>> > > > > > at java.net
> >>> > > > > >
> >>> > >
> >>> .AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> >>> > > > > > at java.net
> >>> > > > > >
> >>> .AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> >>> > > > > > at
> >>> > > > > > java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> >>> > > > > > at java.net.Socket.connect(Socket.java:607) at
> >>> > > > > > java.net.Socket.connect(Socket.java:556) at
> >>> > > > > > java.net.Socket.<init>(Socket.java:452) at
> >>> > > > > > java.net.Socket.<init>(Socket.java:262)
> >>> > > > > >
> >>> > > > > > Some error is logged to /var/log/messages: Oct 30
> >>> > > > > > 05:26:50 home server[65722]: WARNING: Exception
> >>> processing
> >>> > > realm
> >>> > > > > > [com.netscape.cms.tomcat.ProxyRealm@5647a92b]
> >>> > > > > > background
> >>> process
> >>> > > > > > Oct 30 05:26:50 home server[65722]: javax.ws.rs
> >>> > > > > > .ServiceUnavailableException: Subsystem
> >>> unavailable
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> com.netscape.cms.tomcat.ProxyRealm.backgroundProcess(ProxyRealm.java:142)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase.backgroundProcess(ContainerBase.java:1137)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5566)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1353)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1357)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1335)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > >
> >>> > >
> >>> org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
> >>> > > > > > Oct 30 05:26:50 home server[65722]: #011at
> >>> > > > > > java.lang.Thread.run(Thread.java:748)
> >>> > > > > >
> >>> > > > > > I am able to connect to my ldap server port 636 with
> >>> > > > > > TLS
> >>> without
> >>> > > problem.
> >>> > > > > >
> >>> > > > > > Thanks, Patrick
> >>> > > > > >
> >>> > > > > > _______________________________________________
> >>> > > > > > FreeIPA-users mailing list --
> >>> freeipa-users@lists.fedorahosted.org
> >>> > > > > > To unsubscribe send an email to
> >>> > > > > > freeipa-users-leave@lists.fedorahosted.org Fedora
> >>> > > > > > Code of Conduct:
> >>> > > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> > > > > > List Guidelines:
> >>> > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> > > > > > List Archives:
> >>> > > > > >
> >>> > >
> >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >>> > > > > >
> >>> > > > >
> >>> > > > >
> >>> > > > > _______________________________________________
> >>> > > > > FreeIPA-users mailing list --
> >>> freeipa-users@lists.fedorahosted.org
> >>> > > > > To unsubscribe send an email to
> >>> > > freeipa-users-leave@lists.fedorahosted.org
> >>> > > > > Fedora Code of Conduct:
> >>> > > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> > > > > List Guidelines:
> >>> > > https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> > > > > List Archives:
> >>> > > > >
> >>> > >
> >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >>> > > > >
> >>> > > > _______________________________________________
> >>> > > > FreeIPA-users mailing list --
> >>> > > > freeipa-users@lists.fedorahosted.org To unsubscribe send
> >>> > > > an email to
> >>> > > freeipa-users-leave@lists.fedorahosted.org
> >>> > > > Fedora Code of Conduct:
> >>> > > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> > > > List Guidelines:
> >>> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> > > > List Archives:
> >>> > > >
> >>> > >
> >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >>> > > >
> >>> > >
> >>> > >
> >>>
> >>>
> >>> > _______________________________________________
> >>> > FreeIPA-users mailing list --
> >>> > freeipa-users@lists.fedorahosted.org To unsubscribe send an
> >>> > email to
> >>> freeipa-users-leave@lists.fedorahosted.org
> >>> > Fedora Code of Conduct:
> >>> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> >>> > List Guidelines:
> >>> https://fedoraproject.org/wiki/Mailing_list_guidelines
> >>> > List Archives:
> >>> https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org
> >>>
> >>>

> _______________________________________________
> FreeIPA-users mailing list -- freeipa-users@lists.fedorahosted.org
> To unsubscribe send an email to freeipa-users-leave@lists.fedorahosted.org
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org