Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander < alexander_nazarenko@harvard.edu> wrote:
Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497 [2] https://github.com/389ds/389-ds-base/issues/5440 [3] https://github.com/389ds/389-ds-base/issues/5565
best regards thierry
On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander alexander_nazarenko@harvard.edu wrote:
Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems. Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications. Currently we use system controls to restrict the memory usage. My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Casey Feskens cfeskens@willamette.edu Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950
389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Hi,
I did some rapid tests around password update and memory consumption was stable. Did you identify what kind of operation that triggered the growth. You may use [1] to setup the instance with valgrind
[1] https://www.port389.org/docs/389ds/FAQ/faq.html#sts=Debugging%20Memory%20Gro...
regards thierry
On 4/17/23 09:35, Thierry Bordaz wrote:
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497 [2] https://github.com/389ds/389-ds-base/issues/5440 [3] https://github.com/389ds/389-ds-base/issues/5565
best regards thierry
On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander alexander_nazarenko@harvard.edu wrote:
Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems. Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications. Currently we use system controls to restrict the memory usage. My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
--
Casey Feskens cfeskens@willamette.edu Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950
389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Thank you, Thierry,
the operation that is affecting the memory is a simple query: (objectclass=person), which affects every account. This is a standard query we used to use daily for a couple of years with no issues, until lately. The password scheme we use is SSHA512, did not change (I am not seeing PBKDF2-SHA512 option via Apache studio). For 5440, we use memberof a lot, and with “uniqueMember” and “member” membership attributes, it is updated every day on every account.
Shortly I intend use valgrind, as you suggest below.
- Alex
From: Thierry Bordaz tbordaz@redhat.com Reply-To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Date: Monday, April 17, 2023 at 8:43 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, Casey Feskens cfeskens@willamette.edu Subject: [389-users] Re: 389 DS memory growth
Hi,
I did some rapid tests around password update and memory consumption was stable. Did you identify what kind of operation that triggered the growth. You may use [1] to setup the instance with valgrind
[1] https://www.port389.org/docs/389ds/FAQ/faq.html#sts=Debugging%20Memory%20Gro...https://urldefense.proofpoint.com/v2/url?u=https-3A__www.port389.org_docs_389ds_FAQ_faq.html-23sts-3DDebugging-2520Memory-2520Growth_Invalid-2520Access-2520with-25C2-25A0Valgrind&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=s9U5tG4HfYQq-dJ-xHztuL1FPNnaFJAHl-YnKnNuE1k&e=
regards thierry On 4/17/23 09:35, Thierry Bordaz wrote:
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5497&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=MpwdLaS59UHC-JIiHf6_uCVaKtqqvr-R7drVfv9ax5w&e= [2] https://github.com/389ds/389-ds-base/issues/5440https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5440&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=ARXK0Rgcfx0-AVGTM_thOxn3E-NQ7l4MOFq6MGAF1Xs&e= [3] https://github.com/389ds/389-ds-base/issues/5565https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5565&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=8fL_cxRPvq28TAFAhHHcDrj8aWS63qqm91do6xsRqZU&e=
best regards thierry On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=AtlXP_1gsj1ESSRu6iB9KtTDO5Xrh8CSsQLcObGpo4I&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=-b9HgZHFkC0WE94miOFJ3BflSTq4koyFD1dsyi8LxGw&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=L_qNtf4975Z4GPEFWXAFDPtQEos6hQRFrxlkhOJooKU&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=1iYbKOMfPs2V2Woc8zDXOWFVdN_caOqB0jWhXixAoNE&e= -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 ---------------------------------------------
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=AtlXP_1gsj1ESSRu6iB9KtTDO5Xrh8CSsQLcObGpo4I&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=-b9HgZHFkC0WE94miOFJ3BflSTq4koyFD1dsyi8LxGw&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=L_qNtf4975Z4GPEFWXAFDPtQEos6hQRFrxlkhOJooKU&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lV33tMVSt7aDutaHub0AvhxHB8vvDAY60KCUsK1VnEsU7VGfM8lmeSqHscrn6X7t&s=1iYbKOMfPs2V2Woc8zDXOWFVdN_caOqB0jWhXixAoNE&e=
Hi, Thierry,
Where can I see the relation between 389 DS 7.9.* versions and 1.3.* versions of 389-ds-base to keep track of the expected changes?
Thank you, - Alex
From: Thierry Bordaz tbordaz@redhat.com Reply-To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Date: Monday, April 17, 2023 at 3:36 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, Casey Feskens cfeskens@willamette.edu Subject: [389-users] Re: 389 DS memory growth
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5497&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=cVczr8Dfyui60txfEX3PQi22OpsRX9x9_26Ocpvir-U&e= [2] https://github.com/389ds/389-ds-base/issues/5440https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5440&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=Qb6moHqCALNzHsUVhRWWwA_v5MiLdLFio2T3DzrPPxg&e= [3] https://github.com/389ds/389-ds-base/issues/5565https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5565&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=pQ65LCATfGAb6fkaeBzPlxzn7HBNFYJXeT2wCUtwrgc&e=
best regards thierry On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e= -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 ---------------------------------------------
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e=
On 4/21/23 2:18 PM, Nazarenko, Alexander wrote:
Hi, Thierry,
Where can I see the relation between 389 DS 7.9.* versions and 1.3.* versions of 389-ds-base to keep track of the expected changes?
This is comparing upstream verses downstream. It's not so straightforward:
Downstream (search for 389-ds-base) on https://brewweb.engineering.redhat.com/brew/
For example; https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2386883
Upstream you can do git log on the 1.3.x tags:
$ git clone https://github.com/389ds/389-ds-base.git ; cd 389-ds-base
$ git checkout 389-ds-base-1.3.11.1
$ git log --oneline
HTH, Mark
Thank you,
- Alex
*From: *Thierry Bordaz tbordaz@redhat.com *Reply-To: *"General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org *Date: *Monday, April 17, 2023 at 3:36 AM *To: *"General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, Casey Feskens cfeskens@willamette.edu *Subject: *[389-users] Re: 389 DS memory growth
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5497&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=cVczr8Dfyui60txfEX3PQi22OpsRX9x9_26Ocpvir-U&e= [2] https://github.com/389ds/389-ds-base/issues/5440 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5440&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=Qb6moHqCALNzHsUVhRWWwA_v5MiLdLFio2T3DzrPPxg&e= [3] https://github.com/389ds/389-ds-base/issues/5565 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5565&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=pQ65LCATfGAb6fkaeBzPlxzn7HBNFYJXeT2wCUtwrgc&e=
best regards thierry
On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change. On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems. Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications. Currently we use system controls to restrict the memory usage. My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e=> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines <https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e=> List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e=> Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue <https://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e=> -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 --------------------------------------------- _______________________________________________ 389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e=> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines <https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e=> List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e=> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue <https://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e=>
389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
Thank you, it helps, much appreciated.
- Alex
From: Mark Reynolds mareynol@redhat.com Date: Friday, April 21, 2023 at 5:19 PM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, "Nazarenko, Alexander" alexander_nazarenko@harvard.edu, Casey Feskens cfeskens@willamette.edu Subject: Re: [389-users] Re: 389 DS memory growth
On 4/21/23 2:18 PM, Nazarenko, Alexander wrote: Hi, Thierry,
Where can I see the relation between 389 DS 7.9.* versions and 1.3.* versions of 389-ds-base to keep track of the expected changes?
This is comparing upstream verses downstream. It's not so straightforward:
Downstream (search for 389-ds-base) on https://brewweb.engineering.redhat.com/brew/https://urldefense.proofpoint.com/v2/url?u=https-3A__brewweb.engineering.redhat.com_brew_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=fWgBVRckWBooYTJ8FgR2hkXUSx7SgQMbZJdQBlZFDCc&e=
For example; https://brewweb.engineering.redhat.com/brew/buildinfo?buildID=2386883https://urldefense.proofpoint.com/v2/url?u=https-3A__brewweb.engineering.redhat.com_brew_buildinfo-3FbuildID-3D2386883&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=RhqqokRYKJohaYLvqU_6NmSOEyZ4el9FPZFAAt3iyuE&e=
Upstream you can do git log on the 1.3.x tags:
$ git clone https://github.com/389ds/389-ds-base.githttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase.git&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=naAuDVTbMZiNfc6N4DIZHXdYpd-vGqulzjzfKzd9fiw&e= ; cd 389-ds-base
$ git checkout 389-ds-base-1.3.11.1
$ git log --oneline
HTH, Mark
Thank you, - Alex
From: Thierry Bordaz tbordaz@redhat.commailto:tbordaz@redhat.com Reply-To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org Date: Monday, April 17, 2023 at 3:36 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org, Casey Feskens cfeskens@willamette.edumailto:cfeskens@willamette.edu Subject: [389-users] Re: 389 DS memory growth
Hi,
Thanks for raising this issue. Actually the version is an upgrade of 389 7.9.18 to 7.9.21. It contains only 3 bug fixes
- 5497: boolean attribute should be case insensitive - 5440: memberof can be slow when multiple membership attribute are defined - 5565: support of PBKDF2-SHA512 in 7.9
The usual option would be to use valgrind to debug the leak. Because of the limited list of bug we can also try to eliminate candidate. I think the first one looks safe. For 5440, do you use memberof and with how many membership attributes. For 5565, what is your default password storage scheme ? if it is PBKDF2-SHA512, could you set it to PBKDF2-SHA256 and monitor memory consumption ?
[1] https://github.com/389ds/389-ds-base/issues/5497https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5497&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=cVczr8Dfyui60txfEX3PQi22OpsRX9x9_26Ocpvir-U&e= [2] https://github.com/389ds/389-ds-base/issues/5440https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5440&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=Qb6moHqCALNzHsUVhRWWwA_v5MiLdLFio2T3DzrPPxg&e= [3] https://github.com/389ds/389-ds-base/issues/5565https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_389ds_389-2Dds-2Dbase_issues_5565&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=pQ65LCATfGAb6fkaeBzPlxzn7HBNFYJXeT2wCUtwrgc&e=
best regards thierry On 4/17/23 05:38, Casey Feskens wrote:
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e= -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 ---------------------------------------------
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=g5SN0FtjYsArHtitOyspKERV5acf16K9E2tZHPowT58&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=rMifDFHItINAcxL67b7PH-No8b2SqbZ-dtXrlWwmPdg&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=DYfnGKiEMCTuxgJKy7ALpUfQSG8l8DK4onUFBoVJBCU&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=L8I6tptCD6vpEO6wL9Wr1GAH1vrc-ntl1v0Yv3ZkeDhnYN9G4X7dKyatoHufsvhj&s=a6Y-uc-fArhcNYCwmx0Qv4ZBvK2aRrflPNnAF6txnVg&e=
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=e8mCqrms_mmCdLU3a8ulCt8VHKWDXbAi_hNqqf5F2OI&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=PJM4Qe4fBFCtHsfXRftY-DyUJXsrCMOjGQrhAHxKTvI&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=Ihx19M-P4inMCHkIqnOL_PPNZSNThwum9XbZzD0ZfoE&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=-jrzKMkV37hMGXJkxUbfwBnxN7nBeirbfYFeTrMVZfBSw8ttA0cGq0u3Nt2ycnwY&s=2NDkLblmOt2ywqa61y3677cM6C3QyFfH9KWlPmNPVl4&e=
--
Directory Server Development Team
Thank you, Casey,
have you seen the reply by Thierry about probable causes?
Alexander B. Nazarenko, PhD IAM Services | Technology Partner Services Harvard University Information Technology P: 617-496-7150 | M: 617-803-3851
From: Casey Feskens cfeskens@willamette.edu Reply-To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Date: Sunday, April 16, 2023 at 11:39 PM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Subject: [389-users] Re: 389 DS memory growth
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=LISOtgM-5RoIa9CpfJHGcyKVVxyCfu0Scn4iwe7Gpl8&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=_VPhfFXkMtVPLa4z_oVkDwOSUqTLNYEqlfeNJX5AEhk&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=z3VegITLkhxVgpDM88zNqGmKjpLbeVRQX9vvDAqgA_M&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=DNxG9PzA2v-jLuAYl3qP64wC1OZCs3MGnvVV2N2eZ9Q&e= -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 ---------------------------------------------
Yes,
I haven't gotten to debugging yet, but we are not using PBKDF2-SHA512 and aren't routinely using memberof in our own queries as we aren't currently maintaining much in the way of group memberships in this instance. I do believe that memory growth has happened in line with batch jobs doing large queries.
On Mon, Apr 17, 2023 at 12:38 PM Nazarenko, Alexander < alexander_nazarenko@harvard.edu> wrote:
Thank you, Casey,
have you seen the reply by Thierry about probable causes?
*Alexander B. Nazarenko, PhD*
*IAM Services* | Technology Partner Services
*Harvard University Information Technology*
*P**: *617-496-7150* | **M**:* 617-803-3851
*From: *Casey Feskens cfeskens@willamette.edu *Reply-To: *"General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org *Date: *Sunday, April 16, 2023 at 11:39 PM *To: *"General discussion list for the 389 Directory server project." < 389-users@lists.fedoraproject.org> *Subject: *[389-users] Re: 389 DS memory growth
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander < alexander_nazarenko@harvard.edu> wrote:
Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=LISOtgM-5RoIa9CpfJHGcyKVVxyCfu0Scn4iwe7Gpl8&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=_VPhfFXkMtVPLa4z_oVkDwOSUqTLNYEqlfeNJX5AEhk&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=z3VegITLkhxVgpDM88zNqGmKjpLbeVRQX9vvDAqgA_M&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue https://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=DNxG9PzA2v-jLuAYl3qP64wC1OZCs3MGnvVV2N2eZ9Q&e=
--
Casey Feskens cfeskens@willamette.edu Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950
389-users mailing list -- 389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
This is our experience too with 1.3.11*
From: Casey Feskens cfeskens@willamette.edu Reply-To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Date: Tuesday, April 18, 2023 at 12:06 PM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Subject: [389-users] Re: 389 DS memory growth
Yes,
I haven't gotten to debugging yet, but we are not using PBKDF2-SHA512 and aren't routinely using memberof in our own queries as we aren't currently maintaining much in the way of group memberships in this instance. I do believe that memory growth has happened in line with batch jobs doing large queries.
On Mon, Apr 17, 2023 at 12:38 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Thank you, Casey,
have you seen the reply by Thierry about probable causes?
Alexander B. Nazarenko, PhD IAM Services | Technology Partner Services Harvard University Information Technology P: 617-496-7150 | M: 617-803-3851
From: Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Reply-To: "General discussion list for the 389 Directory server project." <389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org> Date: Sunday, April 16, 2023 at 11:39 PM To: "General discussion list for the 389 Directory server project." <389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org> Subject: [389-users] Re: 389 DS memory growth
We’ve been experiencing similar memory growth. I’ve had to quadruple RAM on our ldap hosts, but things seem stable there. Still unsure what the cause is. Glad to hear at least that someone else is seeing the same issue, so I can perhaps rule out an environmental change.
On Sun, Apr 16, 2023 at 6:07 PM Nazarenko, Alexander <alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu> wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=LISOtgM-5RoIa9CpfJHGcyKVVxyCfu0Scn4iwe7Gpl8&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=_VPhfFXkMtVPLa4z_oVkDwOSUqTLNYEqlfeNJX5AEhk&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=z3VegITLkhxVgpDM88zNqGmKjpLbeVRQX9vvDAqgA_M&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=VahkHqwFG118mGcMEtNQ5EabINQr-fub-UhjexRMHVSrL-kDDmdr7MGuaQkFzMBW&s=DNxG9PzA2v-jLuAYl3qP64wC1OZCs3MGnvVV2N2eZ9Q&e= -- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 --------------------------------------------- _______________________________________________ 389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=xpawXSTfydKAf5sUvhT1Z7cy7mVezFisNCeTzbythrjvyHiG6DPRX40QrJubxYQ-&s=K6jgGsFfbmY3G-bn1QxS3ylJbzbta-YnojVdl4wbaR4&e= List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=xpawXSTfydKAf5sUvhT1Z7cy7mVezFisNCeTzbythrjvyHiG6DPRX40QrJubxYQ-&s=cRFvTOSbPT8-p8duN92M1u0k9diRJKz_4Ah91k6EgCc&e= List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=xpawXSTfydKAf5sUvhT1Z7cy7mVezFisNCeTzbythrjvyHiG6DPRX40QrJubxYQ-&s=xPxfBfXQz_V4bYDnhgvic_BGvx9CRLWgjwVzWQ3P_S8&e= Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMFaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=xpawXSTfydKAf5sUvhT1Z7cy7mVezFisNCeTzbythrjvyHiG6DPRX40QrJubxYQ-&s=TWi-rR1DpuIYXIcKjRlLV9ubvFBUIB4EwIFGlb001yk&e=
-- --------------------------------------------- Casey Feskens <cfeskens@willamette.edumailto:cfeskens@willamette.edu> Director of Infrastructure Services Willamette Integrated Technology Services Willamette University, Salem, OR Phone: (503) 370-6950 ---------------------------------------------
Hi,
Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one.
On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning.
The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512.
00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ]
This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point.
regards thierry
On 4/17/23 03:07, Nazarenko, Alexander wrote:
Hello colleagues,
On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update.
Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation?
Thanks,
- Alex
389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.... Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue
This is understood, thank you. It is not a big concern for us, as our servers are at least 16Gb. We are not using pbkdf2 either. This is the heap growth above 20Gb (and up) that is the concern, due to queries like (objectclass=person) hiting the server.
At some point in the near future we plan profile a typical server for memory usage, and plan to keep posted.
- Alwes
From: Thierry Bordaz tbordaz@redhat.com Date: Tuesday, April 18, 2023 at 11:47 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, "Nazarenko, Alexander" alexander_nazarenko@harvard.edu Subject: Re: [389-users] 389 DS memory growth
Hi,
Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one.
On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning.
The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512.
00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ]
This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point.
regards thierry On 4/17/23 03:07, Nazarenko, Alexander wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=vA-Y0-9EUfql9PZclFkEUQNYQWbXzkK_wckNoCsb-Rs&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=2Q_fo7HDWuMCaznM6X-ZFtOxykQo_10PZVZX4FGds3k&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=IAJEMJXS6iY8Zy8s59ThNT9cCsKLvelobO7aDShm-EY&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=XiecP4NfR2TV4r64x1e-H1jn4vfDA6V2Bd2syZxuVxk&e=
Thanks for the update.
I failed to reproduced any significant growth with groups(100)/members(1000) provisioning. The same with searches on person returning 1000 person entries (bound as DM). We will wait for your profiling info.
regards Theirry
On 4/18/23 18:12, Nazarenko, Alexander wrote:
This is understood, thank you. It is not a big concern for us, as our servers are at least 16Gb.
We are not using pbkdf2 either.
This is the heap growth above 20Gb (and up) that is the concern, due to queries like (objectclass=person) hiting the server.
At some point in the near future we plan profile a typical server for memory usage, and plan to keep posted.
*- Alwes*
*From: *Thierry Bordaz tbordaz@redhat.com *Date: *Tuesday, April 18, 2023 at 11:47 AM *To: *"General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org, "Nazarenko, Alexander" alexander_nazarenko@harvard.edu *Subject: *Re: [389-users] 389 DS memory growth
Hi,
Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one.
On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning.
The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512.
00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ]
This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point.
regards thierry
On 4/17/23 03:07, Nazarenko, Alexander wrote:
Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems. Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications. Currently we use system controls to restrict the memory usage. My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex _______________________________________________ 389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=vA-Y0-9EUfql9PZclFkEUQNYQWbXzkK_wckNoCsb-Rs&e=> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines <https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=2Q_fo7HDWuMCaznM6X-ZFtOxykQo_10PZVZX4FGds3k&e=> List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=IAJEMJXS6iY8Zy8s59ThNT9cCsKLvelobO7aDShm-EY&e=> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue <https://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=XiecP4NfR2TV4r64x1e-H1jn4vfDA6V2Bd2syZxuVxk&e=>
This does not contradict to what we see, as noticeable growth starts around the first couple of 100 000 results, and worsens with more.
Thanks, - Alex
From: Thierry Bordaz tbordaz@redhat.com Date: Tuesday, April 18, 2023 at 12:37 PM To: "Nazarenko, Alexander" alexander_nazarenko@harvard.edu, "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Subject: Re: [389-users] 389 DS memory growth
Thanks for the update.
I failed to reproduced any significant growth with groups(100)/members(1000) provisioning. The same with searches on person returning 1000 person entries (bound as DM). We will wait for your profiling info.
regards Theirry On 4/18/23 18:12, Nazarenko, Alexander wrote: This is understood, thank you. It is not a big concern for us, as our servers are at least 16Gb. We are not using pbkdf2 either. This is the heap growth above 20Gb (and up) that is the concern, due to queries like (objectclass=person) hiting the server.
At some point in the near future we plan profile a typical server for memory usage, and plan to keep posted.
- Alwes
From: Thierry Bordaz tbordaz@redhat.commailto:tbordaz@redhat.com Date: Tuesday, April 18, 2023 at 11:47 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org, "Nazarenko, Alexander" alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu Subject: Re: [389-users] 389 DS memory growth
Hi,
Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one.
On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning.
The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512.
00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ]
This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point.
regards thierry On 4/17/23 03:07, Nazarenko, Alexander wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=vA-Y0-9EUfql9PZclFkEUQNYQWbXzkK_wckNoCsb-Rs&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=2Q_fo7HDWuMCaznM6X-ZFtOxykQo_10PZVZX4FGds3k&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=IAJEMJXS6iY8Zy8s59ThNT9cCsKLvelobO7aDShm-EY&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=XiecP4NfR2TV4r64x1e-H1jn4vfDA6V2Bd2syZxuVxk&e=
Hi, Theirry,
We have followed the directions to investigate a memory leak which occurs as we reach many hundreds of thousands of entries, and got the log file, which shows some warnings: ==26735== Warning: set address range perms: large range [0x59eaf000, 0xb3067000) (defined) ==26735== Thread 18: ==26735== Conditional jump or move depends on uninitialized value(s) … ==26735== Thread 30: ==26735== Syscall param pwrite64(buf) points to uninitialised byte(s) ==26735== at 0x7998023: ??? (in /usr/lib64/libpthread-2.17.so)
and similar strings. Memory was running to 90% of 16G servers, and never comes back.
Thoughts, suggestions are much appreciated.
- Alex
From: Thierry Bordaz tbordaz@redhat.com Date: Tuesday, April 18, 2023 at 12:37 PM To: "Nazarenko, Alexander" alexander_nazarenko@harvard.edu, "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org Subject: Re: [389-users] 389 DS memory growth
Thanks for the update.
I failed to reproduced any significant growth with groups(100)/members(1000) provisioning. The same with searches on person returning 1000 person entries (bound as DM). We will wait for your profiling info.
regards Theirry On 4/18/23 18:12, Nazarenko, Alexander wrote: This is understood, thank you. It is not a big concern for us, as our servers are at least 16Gb. We are not using pbkdf2 either. This is the heap growth above 20Gb (and up) that is the concern, due to queries like (objectclass=person) hiting the server.
At some point in the near future we plan profile a typical server for memory usage, and plan to keep posted.
- Alwes
From: Thierry Bordaz tbordaz@redhat.commailto:tbordaz@redhat.com Date: Tuesday, April 18, 2023 at 11:47 AM To: "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org, "Nazarenko, Alexander" alexander_nazarenko@harvard.edumailto:alexander_nazarenko@harvard.edu Subject: Re: [389-users] 389 DS memory growth
Hi,
Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one.
On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning.
The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512.
00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ]
This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point.
regards thierry On 4/17/23 03:07, Nazarenko, Alexander wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems.
Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications.
Currently we use system controls to restrict the memory usage.
My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex
_______________________________________________
389-users mailing list -- 389-users@lists.fedoraproject.orgmailto:389-users@lists.fedoraproject.org
To unsubscribe send an email to 389-users-leave@lists.fedoraproject.orgmailto:389-users-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=vA-Y0-9EUfql9PZclFkEUQNYQWbXzkK_wckNoCsb-Rs&e=
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelineshttps://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=2Q_fo7HDWuMCaznM6X-ZFtOxykQo_10PZVZX4FGds3k&e=
List Archives: https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject....https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=IAJEMJXS6iY8Zy8s59ThNT9cCsKLvelobO7aDShm-EY&e=
Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issuehttps://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=XiecP4NfR2TV4r64x1e-H1jn4vfDA6V2Bd2syZxuVxk&e=
Hi Alexander,
Memory footprint is larger with valgrind. You may reduce cache tuning to let your instance fit on the server. Disable autoh-tune (nsslapd-cache-autosize=0) and reduce nsslapd-cachememsize and nsslapd-dbcachesize.
regards Thierry
On 4/27/23 02:05, Nazarenko, Alexander wrote:
Hi, Theirry,
We have followed the directions to investigate a memory leak which occurs as we reach many hundreds of thousands of entries, and got the log file, which shows some warnings:
==26735== Warning: set address range perms: large range [0x59eaf000, 0xb3067000) (defined)
==26735== Thread 18:
==26735== Conditional jump or move depends on uninitialized value(s)
…
==26735== Thread 30:
==26735== Syscall param pwrite64(buf) points to uninitialised byte(s)
==26735== at 0x7998023: ??? (in /usr/lib64/libpthread-2.17.so)
and similar strings. Memory was running to 90% of 16G servers, and never comes back.
Thoughts, suggestions are much appreciated.
- Alex
*From: *Thierry Bordaz tbordaz@redhat.com *Date: *Tuesday, April 18, 2023 at 12:37 PM *To: *"Nazarenko, Alexander" alexander_nazarenko@harvard.edu, "General discussion list for the 389 Directory server project." 389-users@lists.fedoraproject.org *Subject: *Re: [389-users] 389 DS memory growth
Thanks for the update.
I failed to reproduced any significant growth with groups(100)/members(1000) provisioning. The same with searches on person returning 1000 person entries (bound as DM). We will wait for your profiling info.
regards Theirry
On 4/18/23 18:12, Nazarenko, Alexander wrote:
This is understood, thank you. It is not a big concern for us, as our servers are at least 16Gb. We are not using pbkdf2 either. This is the heap growth above 20Gb (and up) that is the concern, due to queries like (objectclass=person) hiting the server. At some point in the near future we plan profile a typical server for memory usage, and plan to keep posted. *- Alwes* *From: *Thierry Bordaz <tbordaz@redhat.com> <mailto:tbordaz@redhat.com> *Date: *Tuesday, April 18, 2023 at 11:47 AM *To: *"General discussion list for the 389 Directory server project." <389-users@lists.fedoraproject.org> <mailto:389-users@lists.fedoraproject.org>, "Nazarenko, Alexander" <alexander_nazarenko@harvard.edu> <mailto:alexander_nazarenko@harvard.edu> *Subject: *Re: [389-users] 389 DS memory growth Hi, Note that the initial memory footprint of an instance 1.3.11 is larger that an 1.3.10 one. On RHEL 7.9 2Gb VM, an instance 1.3.11 is 1Gb while 1.3.10 is 0.5Gb. Instances have the same DS tuning. The difference comes from extra chunks of anonymous memory (heap) that are possibly related to the new rust plugin handling pbkdf2_sha512. 00007ffb0812e000 64328 0 0 ----- [ anon ] 00007ffb0c000000 1204 1060 1060 rw--- [ anon ] 00007ffb0c12d000 64332 0 0 ----- [ anon ] 00007ffb10000000 1028 1028 1028 rw--- [ anon ] 00007ffb10101000 64508 0 0 ----- [ anon ] 00007ffb14000000 1020 1020 1020 rw--- [ anon ] 00007ffb140ff000 64516 0 0 ----- [ anon ] 00007ffb18000000 1024 1024 1024 rw--- [ anon ] 00007ffb18100000 64512 0 0 ----- [ anon ] 00007ffb1c000000 1044 1044 1044 rw--- [ anon ] 00007ffb1c105000 64492 0 0 ----- [ anon ] 00007ffb20000000 540 540 540 rw--- [ anon ] 00007ffb20087000 64996 0 0 ----- [ anon ] 00007ffb271ce000 4 0 0 ----- [ anon ] This is just the initial memory footprint and does not explain regular growth. Thanks to progier who raised that point. regards thierry On 4/17/23 03:07, Nazarenko, Alexander wrote: Hello colleagues, On March 22nd we updated the 389-ds-base.x86_64 and 389-ds-base-libs.x86_64 packages on our eight RHEL 7.9 production servers from version 1.3.10.2-17.el7_9 to version 1.3.11.1-1.el7_9. We also updated the kernel from kernel 3.10.0-1160.80.1.el7.x86_64 to kernel-3.10.0-1160.88.1.el7.x86_64 during the same update. Approximately 12 days later, on April 3rd, all the hosts started exhibiting memory growth issues whereby the “slapd” process was using over 90% of the available system memory of 32GB, which was NOT happening for a couple of years prior to applying any of the available package updates on the systems. Two of the eight hosts act as Primaries (formerly referred to as masters), while 6 of the hosts act as read-only replicas. Three of the read-only replicas are used by our authorization system while the other three read-only replicas are used by customer-based applications. Currently we use system controls to restrict the memory usage. My question is whether this is something that other users also experience, and what is the recommended way to stabilize the DS servers in this type of situation? Thanks, - Alex _______________________________________________ 389-users mailing list --389-users@lists.fedoraproject.org To unsubscribe send an email to389-users-leave@lists.fedoraproject.org Fedora Code of Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.fedoraproject.org_en-2DUS_project_code-2Dof-2Dconduct_&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=vA-Y0-9EUfql9PZclFkEUQNYQWbXzkK_wckNoCsb-Rs&e=> List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines <https://urldefense.proofpoint.com/v2/url?u=https-3A__fedoraproject.org_wiki_Mailing-5Flist-5Fguidelines&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=2Q_fo7HDWuMCaznM6X-ZFtOxykQo_10PZVZX4FGds3k&e=> List Archives:https://lists.fedoraproject.org/archives/list/389-users@lists.fedoraproject.org <https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.fedoraproject.org_archives_list_389-2Dusers-40lists.fedoraproject.org&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=IAJEMJXS6iY8Zy8s59ThNT9cCsKLvelobO7aDShm-EY&e=> Do not reply to spam, report it:https://pagure.io/fedora-infrastructure/new_issue <https://urldefense.proofpoint.com/v2/url?u=https-3A__pagure.io_fedora-2Dinfrastructure_new-5Fissue&d=DwMDaQ&c=WO-RGvefibhHBZq3fL85hQ&r=uiyPR4nhbOiFgJxO8FlFxvLTOA66849EeL0Dl9-gcSY&m=lhlOLw41Ef61rYH2u6A-2OZLJIGjWUJyMvejNbxxpIlCBNzk8EkZ_ZDJ-b16xD6B&s=XiecP4NfR2TV4r64x1e-H1jn4vfDA6V2Bd2syZxuVxk&e=>
389-users@lists.fedoraproject.org