Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
Hi,
we experience still the same Problems. The OOM Killer kills the ns-slapd process regulary.
Versions (CentOS 7)
# rpm -qa | grep 389 389-adminutil-1.1.22-2.el7.x86_64 389-ds-base-libs-1.3.11.1-3.el7_9.x86_64 389-ds-base-devel-1.3.11.1-3.el7_9.x86_64 389-ds-base-1.3.11.1-3.el7_9.x86_64 389-ds-base-snmp-1.3.11.1-3.el7_9.x86_64 389-ds-base-1.3.10.2-8.el7_9.x86_64 389-ds-base-devel-1.3.10.2-8.el7_9.x86_64 389-admin-1.1.46-4.el7.x86_64 389-adminutil-devel-1.1.22-2.el7.x86_64 389-ds-base-libs-1.3.10.2-8.el7_9.x86_64
# uname -r 3.10.0-1160.11.1.el7.x86_64
The behaviour may be related to unindexed searches, since we have another node that has nearly no unindexed searches and behaves more or less as usual.
br
Harald
On 17.04.23 03:07, Nazarenko, Alexander wrote:
Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex
389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Hi,
389 1.3.11-1-3 is the latest version on that branch and is quite widely installed. So far we are not informed of memory leaks on that release. If cache tuning are reasonable vs available memory and you suspect a leak, I would suggest to use valgrind and the suspected unindexed search.
best regards thierry
On 11/13/23 12:51, Harald Strack wrote:
Hi,
we experience still the same Problems. The OOM Killer kills the ns-slapd process regulary.
Versions (CentOS 7)
# rpm -qa | grep 389 389-adminutil-1.1.22-2.el7.x86_64 389-ds-base-libs-1.3.11.1-3.el7_9.x86_64 389-ds-base-devel-1.3.11.1-3.el7_9.x86_64 389-ds-base-1.3.11.1-3.el7_9.x86_64 389-ds-base-snmp-1.3.11.1-3.el7_9.x86_64 389-ds-base-1.3.10.2-8.el7_9.x86_64 389-ds-base-devel-1.3.10.2-8.el7_9.x86_64 389-admin-1.1.46-4.el7.x86_64 389-adminutil-devel-1.1.22-2.el7.x86_64 389-ds-base-libs-1.3.10.2-8.el7_9.x86_64
# uname -r 3.10.0-1160.11.1.el7.x86_64
The behaviour may be related to unindexed searches, since we have another node that has nearly no unindexed searches and behaves more or less as usual.
br
Harald
On 17.04.23 03:07, Nazarenko, Alexander wrote:
Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex
389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
-- Harald Strack Geschäftsführer
ssystems GmbH Kastanienallee 74 10435 Berlin
389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Hi,
caches where in autosizing mode, this seems not to work in our case (?)
grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif nsslapd-cache-autosize: 10 nsslapd-cache-autosize-split: 40
Only when we set
nsslapd-cache-autosize: 0
and reduce all cashes by a factor of about 10 and 100 (see below):
@@ -1521,11 +1521,11 @@ numSubordinates: 4 nsslapd-suffix: cn=changelog nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
dn: cn=config,cn=ldbm database,cn=plugins,cn=config objectClass: top @@ -1539,7 +1539,7 @@ nsslapd-mode: 600 nsslapd-idlistscanlimit: 4000 nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db -nsslapd-dbcachesize: 1342657822 +nsslapd-dbcachesize: 13426578 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db nsslapd-db-durable-transaction: on nsslapd-db-transaction-wait: off @@ -1586,7 +1586,7 @@ numSubordinates: 4 nsslapd-suffix: o=netscaperoot nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot @@ -1600,11 +1600,11 @@ numSubordinates: 10 nsslapd-suffix: dc=fhm,dc=edu nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
the memory consumption keeps stable. The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.
On 13.11.23 13:22, Thierry Bordaz wrote:
Hi,
389 1.3.11-1-3 is the latest version on that branch and is quite widely installed. So far we are not informed of memory leaks on that release. If cache tuning are reasonable vs available memory and you suspect a leak, I would suggest to use valgrind and the suspected unindexed search.
best regards thierry
So previous tuning was
* dbcache 1.3Gb * 3 backends with 130Mb
How large was it ? How many nsslapd-threadnumber ?
if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable that means there are no leak. But caches are very small. What is the available memory on the box ?
best regards Theirry
On 11/16/23 20:19, Harald Strack wrote:
Hi,
caches where in autosizing mode, this seems not to work in our case (?)
grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif nsslapd-cache-autosize: 10 nsslapd-cache-autosize-split: 40
Only when we set
nsslapd-cache-autosize: 0
and reduce all cashes by a factor of about 10 and 100 (see below):
@@ -1521,11 +1521,11 @@ numSubordinates: 4 nsslapd-suffix: cn=changelog nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
dn: cn=config,cn=ldbm database,cn=plugins,cn=config objectClass: top @@ -1539,7 +1539,7 @@ nsslapd-mode: 600 nsslapd-idlistscanlimit: 4000 nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db -nsslapd-dbcachesize: 1342657822 +nsslapd-dbcachesize: 13426578 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db nsslapd-db-durable-transaction: on nsslapd-db-transaction-wait: off @@ -1586,7 +1586,7 @@ numSubordinates: 4 nsslapd-suffix: o=netscaperoot nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot @@ -1600,11 +1600,11 @@ numSubordinates: 10 nsslapd-suffix: dc=fhm,dc=edu nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
the memory consumption keeps stable. The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.
On 13.11.23 13:22, Thierry Bordaz wrote:
Hi,
389 1.3.11-1-3 is the latest version on that branch and is quite widely installed. So far we are not informed of memory leaks on that release. If cache tuning are reasonable vs available memory and you suspect a leak, I would suggest to use valgrind and the suspected unindexed search.
best regards thierry
-- Harald Strack Geschäftsführer
ssystems GmbH Kastanienallee 74 10435 Berlin
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Maybe it is related to our other Problem "[389-devel] Re: Locking Problem when deleting in parallel" in a way that IO Bottlenecks causing the caching to malloc and retry over and over? Just a theory...
On 17.11.23 11:56, Thierry Bordaz wrote:
So previous tuning was
- dbcache 1.3Gb
- 3 backends with 130Mb
How large was it ? How many nsslapd-threadnumber ?
nsslapd-threadnumber: 128
if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable that means there are no leak. But caches are very small. What is the available memory on the box ?
The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.(see below)
best regards Theirry
On 11/16/23 20:19, Harald Strack wrote:
Hi,
caches where in autosizing mode, this seems not to work in our case (?)
grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif nsslapd-cache-autosize: 10 nsslapd-cache-autosize-split: 40
Only when we set
nsslapd-cache-autosize: 0
and reduce all cashes by a factor of about 10 and 100 (see below):
@@ -1521,11 +1521,11 @@ numSubordinates: 4 nsslapd-suffix: cn=changelog nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
dn: cn=config,cn=ldbm database,cn=plugins,cn=config objectClass: top @@ -1539,7 +1539,7 @@ nsslapd-mode: 600 nsslapd-idlistscanlimit: 4000 nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db -nsslapd-dbcachesize: 1342657822 +nsslapd-dbcachesize: 13426578 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db nsslapd-db-durable-transaction: on nsslapd-db-transaction-wait: off @@ -1586,7 +1586,7 @@ numSubordinates: 4 nsslapd-suffix: o=netscaperoot nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot @@ -1600,11 +1600,11 @@ numSubordinates: 10 nsslapd-suffix: dc=fhm,dc=edu nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
the memory consumption keeps stable. The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.
On 13.11.23 13:22, Thierry Bordaz wrote:
Hi,
389 1.3.11-1-3 is the latest version on that branch and is quite widely installed. So far we are not informed of memory leaks on that release. If cache tuning are reasonable vs available memory and you suspect a leak, I would suggest to use valgrind and the suspected unindexed search.
best regards thierry
-- Harald Strack Geschäftsführer
ssystems GmbH Kastanienallee 74 10435 Berlin
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
On 11/17/23 15:46, Harald Strack wrote:
Maybe it is related to our other Problem "[389-devel] Re: Locking Problem when deleting in parallel" in a way that IO Bottlenecks causing the caching to malloc and retry over and over? Just a theory...
Yes possibly
On 17.11.23 11:56, Thierry Bordaz wrote:
So previous tuning was
- dbcache 1.3Gb
- 3 backends with 130Mb
How large was it ? How many nsslapd-threadnumber ?
nsslapd-threadnumber: 128
It looks a bit high to me. Also each thread will consume a minimal amount for memory, so the more threads/workers the bigger memory footprint. A good fit would be close to the number of core-thread (e.g. 16). But it depends on the workload, if you have many long request and you are fear to exhaust the number of workers the 128 looks good.
if with caches reduced by 10 times (130Mb, 3* 13Mb) it remains stable that means there are no leak. But caches are very small. What is the available memory on the box ?
The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.(see below)
best regards Theirry
On 11/16/23 20:19, Harald Strack wrote:
Hi,
caches where in autosizing mode, this seems not to work in our case (?)
grep nsslapd-cache-autosize /etc/dirsrv/slapd-ldap1/dse.ldif nsslapd-cache-autosize: 10 nsslapd-cache-autosize-split: 40
Only when we set
nsslapd-cache-autosize: 0
and reduce all cashes by a factor of about 10 and 100 (see below):
@@ -1521,11 +1521,11 @@ numSubordinates: 4 nsslapd-suffix: cn=changelog nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/changelog -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
dn: cn=config,cn=ldbm database,cn=plugins,cn=config objectClass: top @@ -1539,7 +1539,7 @@ nsslapd-mode: 600 nsslapd-idlistscanlimit: 4000 nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db -nsslapd-dbcachesize: 1342657822 +nsslapd-dbcachesize: 13426578 nsslapd-db-logdirectory: /var/lib/dirsrv/slapd-ldap1/db nsslapd-db-durable-transaction: on nsslapd-db-transaction-wait: off @@ -1586,7 +1586,7 @@ numSubordinates: 4 nsslapd-suffix: o=netscaperoot nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/NetscapeRoot @@ -1600,11 +1600,11 @@ numSubordinates: 10 nsslapd-suffix: dc=fhm,dc=edu nsslapd-cachesize: -1 -nsslapd-cachememsize: 671088640 +nsslapd-cachememsize: 67108864 nsslapd-readonly: off nsslapd-require-index: off nsslapd-directory: /var/lib/dirsrv/slapd-ldap1/db/userRoot -nsslapd-dncachememsize: 134217728 +nsslapd-dncachememsize: 13421772
the memory consumption keeps stable. The machine has 32 GB memory, so the values prefixed with a minus above should easyly fit.
On 13.11.23 13:22, Thierry Bordaz wrote:
Hi,
389 1.3.11-1-3 is the latest version on that branch and is quite widely installed. So far we are not informed of memory leaks on that release. If cache tuning are reasonable vs available memory and you suspect a leak, I would suggest to use valgrind and the suspected unindexed search.
best regards thierry
-- Harald Strack Geschäftsführer
ssystems GmbH Kastanienallee 74 10435 Berlin
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
-- Harald Strack Geschäftsführer
ssystems GmbH Kastanienallee 74 10435 Berlin
-- _______________________________________________ 389-devel mailing list --389-devel@lists.fedoraproject.org To unsubscribe send an email to389-devel-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
389-devel@lists.fedoraproject.org