Hi Jim,

To my understanding anybody can create a Bugzilla ticket. Just let me
know if it doesn't work.

The exception is likely triggered by the content of the database, so if you
could provide the LDAP entry that triggers the error (i.e. cn=268174648,
ou=certificateRepository,ou=ca,o=ipaca) that would be helpful. Feel free to
remove sensitive attribute values, but we need to know what attributes the
entry has.

I'm not sure about the memory leak in ns-slapd. That would be handled by
another team. You might want to open a Bugzilla ticket against 389-ds-base.

--
Endi S. Dewata

On Mon, Apr 26, 2021 at 1:39 PM Jim Richard <jrichard@placeiq.com> wrote:
Hi Endi:

We haven’t done any customizations. I’ve owned this FreeIPA deployment since way back in the CentOS 6.X days, when all the CA stuff lived in a second LDAP instance. Since then there have been numerous upgrades, cleanup of old data, some minor tweaks to things like nsslapd-idletimeout etc. but nothing major.

We have a pretty simple use case, about 800 hosts, 3 nodes, about 30 active users, we mainly use FreeIPA to enforce hosts access, via HBAC and sudo rules, user public ssh keys.

I don’t know if this is related but back in February of this year we upgraded from  4.8.4-7 to 4.8.7-14 and started to see a slow and steady apparent memory leak from ns-slapd:

https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedorahosted.org/message/X7I3N2LZCW2CQBA5YWJ7G6LROQMH7YRP/


Did you guys want me to create the Bugzilla ticket? I’m not sure if I can as a third party. Happy to though.



Jim Richard
System Administrator III
jrichard@placeiq.com | (646) 338-8905 | www.placeiq.com

On Apr 26, 2021, at 9:53 AM, Endi Dewata <edewata@redhat.com> wrote:

Hi,

I have not seen this error before. Was there any customization done to the system? Could you open a Bugzilla ticket against pki-core component on RHEL 8 and provide the steps to reproduce? Also, please include the LDAP entry cn=268174648,ou=certificateRepository,ou=ca,o=ipaca. Thanks.

--
Endi S. Dewata

On Mon, Apr 26, 2021, 7:40 AM Rob Crittenden <rcritten@redhat.com> wrote:
Jim Richard via FreeIPA-users wrote:
> From /var/log/pki/pki-tomcat/ca/debug.2021-04-26.log
>
>
> 2021-04-26 04:13:42 [ajp-nio-127.0.0.1-8009-exec-4] INFO: DBVirtualList: dn: cn=268174647,ou=certificateRepository,ou=ca,o=ipaca
> 2021-04-26 04:13:42 [ajp-nio-127.0.0.1-8009-exec-4] INFO: DBVirtualList: dn: cn=268174648,ou=certificateRepository,ou=ca,o=ipaca
>
> 2021-04-26 04:13:42 [ajp-nio-127.0.0.1-8009-exec-4] SEVERE: Operation Error - netscape.ldap.LDAPException cannot be cast to netscape.ldap.LDAPEntry
> java.lang.ClassCastException: netscape.ldap.LDAPException cannot be cast to netscape.ldap.LDAPEntry
>         at com.netscape.cmscore.dbs.DBVirtualList.getEntries(DBVirtualList.java:477)
>         at com.netscape.cmscore.dbs.DBVirtualList.getPage(DBVirtualList.java:610)
>         at com.netscape.cmscore.dbs.DBVirtualList.getPage(DBVirtualList.java:602)
>         at com.netscape.cmscore.dbs.DBVirtualList.getElementAt(DBVirtualList.java:754)
>         at com.netscape.cmscore.dbs.CertRecordList.getCertRecord(CertRecordList.java:110)
>         at org.dogtagpki.server.ca.rest.CertService.searchCerts(CertService.java:473)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
>         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
>         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
>         at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
>         at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>         at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
>         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
>         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
>         at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
>         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
>         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
>         at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:82)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:428)
>         at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598)
>         at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
>
> 2021-04-26 04:13:42 [ajp-nio-127.0.0.1-8009-exec-4] SEVERE: Unable to search for certificates: java.lang.ClassCastException: netscape.ldap.LDAPException cannot be cast to netscape.ldap.LDAPEntry
> java.lang.RuntimeException: java.lang.ClassCastException: netscape.ldap.LDAPException cannot be cast to netscape.ldap.LDAPEntry
>         at com.netscape.cmscore.dbs.DBVirtualList.getEntries(DBVirtualList.java:523)
>         at com.netscape.cmscore.dbs.DBVirtualList.getPage(DBVirtualList.java:610)
>         at com.netscape.cmscore.dbs.DBVirtualList.getPage(DBVirtualList.java:602)
>         at com.netscape.cmscore.dbs.DBVirtualList.getElementAt(DBVirtualList.java:754)
>         at com.netscape.cmscore.dbs.CertRecordList.getCertRecord(CertRecordList.java:110)
>         at org.dogtagpki.server.ca.rest.CertService.searchCerts(CertService.java:473)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.jboss.resteasy.core.MethodInjectorImpl.invoke(MethodInjectorImpl.java:140)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invokeOnTarget(ResourceMethodInvoker.java:295)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:249)
>         at org.jboss.resteasy.core.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:236)
>         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:406)
>         at org.jboss.resteasy.core.SynchronousDispatcher.invoke(SynchronousDispatcher.java:213)
>         at org.jboss.resteasy.plugins.server.servlet.ServletContainerDispatcher.service(ServletContainerDispatcher.java:228)
>         at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:56)
>         at org.jboss.resteasy.plugins.server.servlet.HttpServletDispatcher.service(HttpServletDispatcher.java:51)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
>         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:170)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:225)
>         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
>         at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:498)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:282)
>         at org.apache.catalina.security.SecurityUtil$1.run(SecurityUtil.java:279)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAsPrivileged(Subject.java:549)
>         at org.apache.catalina.security.SecurityUtil.execute(SecurityUtil.java:314)
>         at org.apache.catalina.security.SecurityUtil.doAsPrivilege(SecurityUtil.java:253)
>         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:191)
>         at org.apache.catalina.core.ApplicationFilterChain.access$000(ApplicationFilterChain.java:47)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:149)
>         at org.apache.catalina.core.ApplicationFilterChain$1.run(ApplicationFilterChain.java:145)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:144)
>         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
>         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
>         at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
>         at com.netscape.cms.tomcat.ExternalAuthenticationValve.invoke(ExternalAuthenticationValve.java:82)
>         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
>         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
>         at org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:678)
>         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
>         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
>         at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:428)
>         at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
>         at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:860)
>         at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1598)
>         at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
>         at java.lang.Thread.run(Thread.java:748)
>
> Caused by: java.lang.ClassCastException: netscape.ldap.LDAPException cannot be cast to netscape.ldap.LDAPEntry
>         at com.netscape.cmscore.dbs.DBVirtualList.getEntries(DBVirtualList.java:477)
>         ... 70 more
>
>
>
>
>
> Current versions are:
>
> CentOS 8:
>
> ipa-client.x86_64                      4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
> ipa-client-common.noarch               4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
> ipa-common.noarch                      4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
> ipa-healthcheck.noarch                 0.4-6.module_el8.3.0+482+9e103aab            @AppStream   
> ipa-healthcheck-core.noarch            0.4-6.module_el8.3.0+482+9e103aab            @AppStream   
> ipa-selinux.noarch                     4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
> ipa-server.x86_64                      4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
> ipa-server-common.noarch               4.8.7-14.module_el8.3.0+698+d6d67052         @appstream   
>
> 389-ds-base.x86_64                     1.4.3.8-6.module_el8.3.0+604+ab7bf9cc        @AppStream   
> 389-ds-base-libs.x86_64                1.4.3.8-6.module_el8.3.0+604+ab7bf9cc        @AppStream
>
> Linux sso-111 4.18.0-240.15.1.el8_3.x86_64 #1 SMP Mon Mar 1 17:16:16 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

cc'ing one of the CA developers.

rob