Looks like using real time scheduler never worked, since we setup the scheduler after droppig privileges.
Fix Makefile so we can build and test sanlock on Fedora 31.
Nir Soffer (3): Makfile: Use PY_VERSION=3 Makefile: %systemd_postun requires arguments now sanlock: Setup priority before dropping privileges
sanlock.spec.in | 6 +++--- src/main.c | 4 ++-- 2 files changed, 5 insertions(+), 5 deletions(-)
The makefile was using 3.6 which is leftover from Fedora 29. Using python3 works with any python 3 version.
Signed-off-by: Nir Soffer nsoffer@redhat.com --- sanlock.spec.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/sanlock.spec.in b/sanlock.spec.in index 087076c..9ae4d8f 100644 --- a/sanlock.spec.in +++ b/sanlock.spec.in @@ -46,7 +46,7 @@ The sanlock daemon manages leases for applications on hosts using shared storage # upstream does not support _smp_mflags CFLAGS=$RPM_OPT_FLAGS make -C wdmd CFLAGS=$RPM_OPT_FLAGS make -C src -CFLAGS=$RPM_OPT_FLAGS make -C python PY_VERSION=3.6 +CFLAGS=$RPM_OPT_FLAGS make -C python PY_VERSION=3 CFLAGS=$RPM_OPT_FLAGS make -C reset
%install @@ -60,7 +60,7 @@ make -C wdmd \ make -C python \ install LIBDIR=%{_libdir} \ DESTDIR=$RPM_BUILD_ROOT \ - PY_VERSION=3.6 + PY_VERSION=3 make -C reset \ install LIBDIR=%{_libdir} \ DESTDIR=$RPM_BUILD_ROOT
The makefile was using 3.6 which is leftover from Fedora 29. Using python3 works with any python 3 version.
+1
Signed-off-by: Nir Soffer nsoffer@redhat.com
sanlock.spec.in | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/sanlock.spec.in b/sanlock.spec.in index 087076c..9ae4d8f 100644 --- a/sanlock.spec.in +++ b/sanlock.spec.in @@ -46,7 +46,7 @@ The sanlock daemon manages leases for applications on hosts using shared storage
# upstream does not support _smp_mflags
CFLAGS=$RPM_OPT_FLAGS make -C wdmd CFLAGS=$RPM_OPT_FLAGS make -C src -CFLAGS=$RPM_OPT_FLAGS make -C python PY_VERSION=3.6 +CFLAGS=$RPM_OPT_FLAGS make -C python PY_VERSION=3 CFLAGS=$RPM_OPT_FLAGS make -C reset
%install @@ -60,7 +60,7 @@ make -C wdmd \ make -C python \ install LIBDIR=%{_libdir} \ DESTDIR=$RPM_BUILD_ROOT \
PY_VERSION=3.6
PY_VERSION=3
make -C reset \ install LIBDIR=%{_libdir} \ DESTDIR=$RPM_BUILD_ROOT -- 2.25.4 _______________________________________________ sanlock-devel mailing list -- sanlock-devel@lists.fedorahosted.org To unsubscribe send an email to sanlock-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sanlock-devel@lists.fedorahost ed.org
Since Fedora 31, %system_postun requires arguments, and fail with this "helpful" message without arguments:
error: This macro requires some arguments
Signed-off-by: Nir Soffer nsoffer@redhat.com --- sanlock.spec.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sanlock.spec.in b/sanlock.spec.in index 9ae4d8f..b14da9b 100644 --- a/sanlock.spec.in +++ b/sanlock.spec.in @@ -99,7 +99,7 @@ getent passwd sanlock > /dev/null || /usr/sbin/useradd \ %systemd_preun wdmd.service sanlock.service
%postun -%systemd_postun +%systemd_postun wdmd.service sanlock.service
%files /usr/lib/systemd/systemd-wdmd
Since Fedora 31, %system_postun requires arguments, and fail with this "helpful" message without arguments:
error: This macro requires some arguments
+1
Signed-off-by: Nir Soffer nsoffer@redhat.com
sanlock.spec.in | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/sanlock.spec.in b/sanlock.spec.in index 9ae4d8f..b14da9b 100644 --- a/sanlock.spec.in +++ b/sanlock.spec.in @@ -99,7 +99,7 @@ getent passwd sanlock > /dev/null || /usr/sbin/useradd \ %systemd_preun wdmd.service sanlock.service
%postun -%systemd_postun +%systemd_postun wdmd.service sanlock.service
%files /usr/lib/systemd/systemd-wdmd -- 2.25.4 _______________________________________________ sanlock-devel mailing list -- sanlock-devel@lists.fedorahosted.org To unsubscribe send an email to sanlock-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/sanlock-devel@lists.fedorahost ed.org
sched_setscheduler() requires root, but we called it after dropping privileges, so it always failed:
2020-02-13 12:34:19 1480 [8866]: sanlock daemon started 3.8.0 host a08359de-225c-4c21-a7d6-3623bb3bd6fb.host4 2020-02-13 12:34:19 1480 [8866]: set scheduler RR|RESET_ON_FORK priority 99 failed: Operation not permitted
Move setup_priority up before we drop privileges.
With this change sanlock runs now with RR scheduler and expected priority:
$ ps -o cmd,cls,rtprio -p 2275 CMD CLS RTPRIO /usr/sbin/sanlock daemon RR 99
Not running with real time scheduler may be the reason we see random failures to write lockspace in oVirt system tests: https://bugzilla.redhat.com/1247135
Signed-off-by: Nir Soffer nsoffer@redhat.com --- src/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main.c b/src/main.c index 8c6eef8..ebc0b11 100644 --- a/src/main.c +++ b/src/main.c @@ -1750,14 +1750,14 @@ static int do_daemon(void)
setup_host_name();
+ setup_priority(); + setup_uid_gid();
uname(&nodename);
log_warn("sanlock daemon started %s host %s (%s)", VERSION, our_host_name_global, nodename.nodename);
- setup_priority(); - rv = thread_pool_create(DEFAULT_MIN_WORKER_THREADS, com.max_worker_threads); if (rv < 0) goto out;
On Sun, May 3, 2020 at 2:35 AM Nir Soffer nirsof@gmail.com wrote:
sched_setscheduler() requires root, but we called it after dropping privileges, so it always failed:
2020-02-13 12:34:19 1480 [8866]: sanlock daemon started 3.8.0 host a08359de-225c-4c21-a7d6-3623bb3bd6fb.host4 2020-02-13 12:34:19 1480 [8866]: set scheduler RR|RESET_ON_FORK priority 99 failed: Operation not permitted
Move setup_priority up before we drop privileges.
With this change sanlock runs now with RR scheduler and expected priority:
$ ps -o cmd,cls,rtprio -p 2275 CMD CLS RTPRIO /usr/sbin/sanlock daemon RR 99
There is one issue, this works on Fedora but not on RHEL. On RHEL the call always fail, and on Fedora, setting priority works even without this change, so this change does not solve the problem.
Looks like the way to use real time scheduler in RHEL 8 is using systemd: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_r...
I'll send another patch.
Not running with real time scheduler may be the reason we see random failures to write lockspace in oVirt system tests: https://bugzilla.redhat.com/1247135
Signed-off-by: Nir Soffer nsoffer@redhat.com
src/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main.c b/src/main.c index 8c6eef8..ebc0b11 100644 --- a/src/main.c +++ b/src/main.c @@ -1750,14 +1750,14 @@ static int do_daemon(void)
setup_host_name();
setup_priority();
setup_uid_gid(); uname(&nodename); log_warn("sanlock daemon started %s host %s (%s)", VERSION, our_host_name_global, nodename.nodename);
setup_priority();
rv = thread_pool_create(DEFAULT_MIN_WORKER_THREADS, com.max_worker_threads); if (rv < 0) goto out;
-- 2.25.4
On Sun, May 3, 2020 at 7:46 PM Nir Soffer nsoffer@redhat.com wrote:
On Sun, May 3, 2020 at 2:35 AM Nir Soffer nirsof@gmail.com wrote:
sched_setscheduler() requires root, but we called it after dropping privileges, so it always failed:
2020-02-13 12:34:19 1480 [8866]: sanlock daemon started 3.8.0 host a08359de-225c-4c21-a7d6-3623bb3bd6fb.host4 2020-02-13 12:34:19 1480 [8866]: set scheduler RR|RESET_ON_FORK priority 99 failed: Operation not permitted
Move setup_priority up before we drop privileges.
With this change sanlock runs now with RR scheduler and expected priority:
$ ps -o cmd,cls,rtprio -p 2275 CMD CLS RTPRIO /usr/sbin/sanlock daemon RR 99
There is one issue, this works on Fedora but not on RHEL. On RHEL the call always fail, and on Fedora, setting priority works even without this change, so this change does not solve the problem.
Looks like the way to use real time scheduler in RHEL 8 is using systemd: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux_for_r...
It does not work, I tried:
diff --git a/init.d/sanlock.service.native b/init.d/sanlock.service.native index f14eccd..53d61e9 100644 --- a/init.d/sanlock.service.native +++ b/init.d/sanlock.service.native @@ -2,6 +2,8 @@ Description=Shared Storage Lease Manager After=syslog.target Wants=wdmd.service +CPUSchedulingPolicy=rr +CPUSchedulingPriority=99
[Service] Type=forking diff --git a/init.d/wdmd.service.native b/init.d/wdmd.service.native index ab0828e..ed6a53d 100644 --- a/init.d/wdmd.service.native +++ b/init.d/wdmd.service.native @@ -1,6 +1,8 @@ [Unit] Description=Watchdog Multiplexing Daemon After=syslog.target +CPUSchedulingPolicy=rr +CPUSchedulingPriority=99
[Service] Type=forking
But system fail to change cpu scheduling:
# systemctl start wdmd Job for wdmd.service failed because the control process exited with error code. See "systemctl status wdmd.service" and "journalctl -xe" for details. [root@host4 ~]# systemctl status wdmd ● wdmd.service - Watchdog Multiplexing Daemon Loaded: loaded (/usr/lib/systemd/system/wdmd.service; disabled; vendor preset: disabled) Active: failed (Result: exit-code) since Sun 2020-05-03 21:19:13 IDT; 7s ago Process: 2379 ExecStartPre=/lib/systemd/systemd-wdmd watchdog-check (code=exited, status=214/SETSCHEDULER) Main PID: 2338 (code=exited, status=0/SUCCESS)
May 03 21:19:13 host4 systemd[1]: Starting Watchdog Multiplexing Daemon... May 03 21:19:13 host4 systemd[1]: wdmd.service: Control process exited, code=exited status=214 May 03 21:19:13 host4 systemd[1]: wdmd.service: Failed with result 'exit-code'. May 03 21:19:13 host4 systemd[1]: Failed to start Watchdog Multiplexing Daemon.
Searching this issue, it seems that this feature is simply not supported by systemd.
I think we need to file RHEL systemd bug and get help from systemd folks about this.
Looking in Fedora 30 system we see:
# ps -eo cmd,cls,rtprio | grep RR | grep -v grep /sbin/multipathd -d -s RR 99 /usr/sbin/sanlock daemon RR 99 /usr/sbin/wdmd RR 99
# ps -eo cmd,cls,rtprio | grep FF | grep -v grep [migration/0] FF 99 [migration/1] FF 99 [watchdogd] FF 99 [irq/24-aerdrv] FF 50 [irq/24-pciehp] FF 50 [irq/25-aerdrv] FF 50 [irq/25-pciehp] FF 50 [irq/26-aerdrv] FF 50 [irq/26-pciehp] FF 50 [irq/27-aerdrv] FF 50 [irq/27-pciehp] FF 50 [irq/28-aerdrv] FF 50 [irq/28-pciehp] FF 50 [irq/29-aerdrv] FF 50 [irq/29-pciehp] FF 50 [irq/30-aerdrv] FF 50 [irq/30-pciehp] FF 50
Looking in other processes on RHEL 8.2, we see:
# ps -eo cmd,cls,rtprio | grep RR | grep -v grep (nothing)
# ps -eo cmd,cls,rtprio | grep FF | grep -v grep [migration/0] FF 99 [watchdog/0] FF 99 [watchdog/1] FF 99 [migration/1] FF 99 [watchdogd] FF 99 [irq/24-aerdrv] FF 50 [irq/24-pciehp] FF 50 [irq/25-aerdrv] FF 50 [irq/25-pciehp] FF 50 [irq/26-aerdrv] FF 50 [irq/26-pciehp] FF 50 [irq/27-aerdrv] FF 50 [irq/27-pciehp] FF 50 [irq/28-aerdrv] FF 50 [irq/28-pciehp] FF 50 [irq/29-aerdrv] FF 50 [irq/29-pciehp] FF 50 [irq/30-aerdrv] FF 50 [irq/30-pciehp] FF 50
multipathd runs as:
# ps -eo cmd,cls,rtprio | grep multipathd | grep -v grep /sbin/multipathd -d -s TS -
So looks like Ben has the same issue.
So FF (SCHED_FIFO) works.
Should we use FF (SCHED_FIFO) with max priority instead?
We can try to use SCHED_RR, and fallback to SCHED_FIFO if failed.
David, Ben, what do you think?
I'll send another patch.
Not running with real time scheduler may be the reason we see random failures to write lockspace in oVirt system tests: https://bugzilla.redhat.com/1247135
Signed-off-by: Nir Soffer nsoffer@redhat.com
src/main.c | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/src/main.c b/src/main.c index 8c6eef8..ebc0b11 100644 --- a/src/main.c +++ b/src/main.c @@ -1750,14 +1750,14 @@ static int do_daemon(void)
setup_host_name();
setup_priority();
setup_uid_gid(); uname(&nodename); log_warn("sanlock daemon started %s host %s (%s)", VERSION, our_host_name_global, nodename.nodename);
setup_priority();
rv = thread_pool_create(DEFAULT_MIN_WORKER_THREADS, com.max_worker_threads); if (rv < 0) goto out;
-- 2.25.4
sanlock-devel@lists.fedorahosted.org