sanlock-devel November 2022

sanlock-devel@lists.fedorahosted.org

3 participants
9 discussions

[PATCH 0/6] Add github CI
by Nir Soffer 12 Dec '22

12 Dec '22

Replace the broken travis CI configuration with github actions. The new CI revealed some issues with chown(), and some flaky or broken tests with some python versions. The chown() issues are fixed in this change, the tests issues need investigation. Nir Soffer (6): Remove travis CI configuration Log uid and gid when running in unprivileged mode Change rundir ownership only if running as root Don't change socket ownership Add python 3.11 to tox Add github actions based CI .github/workflows/ci.yml | 48 ++++++++++++++++++++++++++++++++++++++++ .travis.yml | 26 ---------------------- src/lockfile.c | 12 +++++----- src/main.c | 10 ++------- tox.ini | 4 ++-- 5 files changed, 59 insertions(+), 41 deletions(-) create mode 100644 .github/workflows/ci.yml delete mode 100644 .travis.yml -- 2.38.1

2 9

[PATCH 0/2] Improve README.dev
by Nir Soffer 12 Dec '22

12 Dec '22

- Improve instructions for creating python virtual environment - Remove stale info about python 2.7 Nir Soffer (2): Add section for creating and using venv Update README.dev for python 3 README.dev | 68 +++++++++++++++++++++++++----------------------------- 1 file changed, 32 insertions(+), 36 deletions(-) -- 2.38.1

2 3

[PATCH 0/3] Improve disk validation
by Nir Soffer 27 Nov '22

27 Nov '22

Fix failing disk validation test since python 3.9, and improve error messages when disk validation fails. The tests succeeds now on python3.6-python3.11: https://github.com/nirs/sanlock/actions/runs/3559586867 Nir Soffer (3): Improve error message if PyObject_Repr failed Improve error messages when validating disks Fix disk validation on python >= 3.9 python/sanlock.c | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) -- 2.38.1

1 3

[PATCH v3 0/4] New patch for updating userstorage package
by Albert Esteve 27 Nov '22

27 Nov '22

Sanlock is using an old version of userstorage in its tests. The package it's been installable through package manager for a while now, and it has been recently updated to its version 0.5.1. So I have created a patch to use the latest version that can be installed from PyPI, and remove the old embedded package. The structure of the test storage folder remains the same. Also added a couple commits to add latest python versions, and a prerrequisite to the documentation that I needed to install to be able to compile. But I can remove these unrelated commits. Albert Esteve (4): README.dev: update prerequisites tox.ini: add py310 to environments userstorage: replace by PyPI userstorage README.dev: update userstorage doc README.dev | 22 +++++-- requirements.txt | 1 + tests/conftest.py | 22 ++++--- tests/storage.py | 161 +++++++--------------------------------------- tox.ini | 5 +- 5 files changed, 55 insertions(+), 156 deletions(-) create mode 100644 requirements.txt -- 2.37.3

2 7

[sanlock] 04/04: README.dev: update userstorage doc
by pagure＠pagure.io 27 Nov '22

27 Nov '22

This is an automated email from the git hooks/post-receive script. nsoffer pushed a commit to branch master in repository sanlock. commit 1b306f0abb173393cb478f09d58a9b0777bc1387 Author: Albert Esteve <aesteve(a)redhat.com> AuthorDate: Wed Nov 9 14:50:57 2022 +0100 README.dev: update userstorage doc Update documentation regarding userstorage setup and teardown. Signed-off-by: Albert Esteve <aesteve(a)redhat.com> Signed-off-by: Nir Soffer <nsoffer(a)redhat.com> --- README.dev | 30 +++++++++++++++++++++++++++--- 1 file changed, 27 insertions(+), 3 deletions(-) diff --git a/README.dev b/README.dev index 1bf1d17..46b0088 100644 --- a/README.dev +++ b/README.dev @@ -54,9 +54,29 @@ Or using environment variable: Testing 4K support ================== -To enable the 4k tests, you need to setup 4k stroage for the tests: +To enable the 4k tests, you need to have userstorage installed. +First, you need to create a virtual environment +(https://docs.python.org/3/library/venv.html), and then install +the userstorage library from the requirements.txt. +This only needs to be done once: - $ python tests/storage.py setup + $ python3 -m venv ~/.venv/sanlock + $ source ~/.venv/sanlock/bin/activate + $ pip install --upgrade pip + $ pip install -r requirements.txt + $ deactivate + +Before running the tests, you need to enter the virtual environment: + + $ source ~/.venv/sanlock/bin/activate + +The shell prompt will change to show the virtual environment name: + + (sanlock) [user@hostname sanlock]$ + +Then, you just need to setup 4k storage for the tests: + + $ userstorage create tests/storage.py This creates two loop devices with 4k sector size; one for testing sanlock with 4k block device, and the other for testing with a filesystem backed by a 4k @@ -64,10 +84,14 @@ block device. To teardown the storage: - $ python tests/storage.py teardown + $ userstorage delete tests/storage.py The script unmounts the filesystem and detaches the loop devices. The storage helper script uses sudo to perform privileged operations. The best way to use it is to setup the environment once at the start of the session, and teardown when you finish. + +To deactivate the virtual environment: + + $ deactivate -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[sanlock] 03/04: userstorage: replace by PyPI userstorage
by pagure＠pagure.io 27 Nov '22

27 Nov '22

This is an automated email from the git hooks/post-receive script. nsoffer pushed a commit to branch master in repository sanlock. commit 38ff68588078af1664787e20256ffba3ae166489 Author: Albert Esteve <aesteve(a)redhat.com> AuthorDate: Wed Nov 9 14:50:56 2022 +0100 userstorage: replace by PyPI userstorage Remove the outdated version of userstorage that is stored locally at tests/storage.py The userstorage library is required to create the storage before running the tests with tox, and also in the test virtual environment. In order to satisfy this, add a requirements.txt to the root folder to obtain the latests userstorage version from PyPI, and add it as a tox dependency for tests. Behaviour for missing storage and folder structure remains as originally intended. Fixture is adapted to new style so that is uses setup and teardown before and after every test, respectively. Signed-off-by: Albert Esteve <aesteve(a)redhat.com> --- requirements.txt | 2 + tests/conftest.py | 22 ++++---- tests/storage.py | 161 ++++++++---------------------------------------------- tox.ini | 1 + 4 files changed, 37 insertions(+), 149 deletions(-) diff --git a/requirements.txt b/requirements.txt new file mode 100644 index 0000000..2dfdbcd --- /dev/null +++ b/requirements.txt @@ -0,0 +1,2 @@ +tox +userstorage>=0.5.1 diff --git a/tests/conftest.py b/tests/conftest.py index b40852b..79c514b 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -9,13 +9,17 @@ Fixtures for sanlock testing. """ from __future__ import absolute_import -import os - import pytest +import userstorage -from . import storage from . import util +# Mark tests with skip if userstorage is missing +userstorage.missing_handler = pytest.skip + +# Requires relative path from tox basedir +BACKENDS = userstorage.load_config("./tests/storage.py").BACKENDS + class SanlockIsRunning(Exception): """ Raised if sanlock running when it should not """ @@ -38,8 +42,8 @@ def sanlock_daemon(): @pytest.fixture(params=[ - pytest.param(storage.BLOCK, id="block"), - pytest.param(storage.FILE, id="file"), + BACKENDS["block"], + BACKENDS["file"], ]) def user_4k_path(request): """ @@ -48,11 +52,9 @@ def user_4k_path(request): If storage is not available, skip the tests. """ - if not os.path.exists(request.param): - pytest.skip( - "user storage available - run 'python tests/strorage.py setup' " - "to enable 4k storage tests") - return request.param + backend = request.param + with backend: + yield backend.path @pytest.fixture diff --git a/tests/storage.py b/tests/storage.py index 0fe6f56..0b12e47 100644 --- a/tests/storage.py +++ b/tests/storage.py @@ -8,145 +8,28 @@ storage - provide storage for sanlock tests. """ -import argparse -import errno -import logging -import os -import subprocess +from userstorage import File, Mount, LoopDevice -BASE_DIR = "/var/tmp/sanlock-storage" -MOUNTPOINT = os.path.join(BASE_DIR, "mnt") - -# For testing sanlock with 4k block device. -LOOP1 = os.path.join(BASE_DIR, "loop1") -BACKING1 = os.path.join(BASE_DIR, "backing1") - -# For testing sanlock with a filesystem backed by 4k block device. -LOOP2 = os.path.join(BASE_DIR, "loop2") -BACKING2 = os.path.join(BASE_DIR, "backing2") - -# Test paths. -BLOCK = LOOP1 -FILE = os.path.join(MOUNTPOINT, "file") - -log = logging.getLogger("storage") - - -def main(): - parser = argparse.ArgumentParser( - description='Storage helper for sanlock tests') - parser.add_argument("command", choices=["setup", "teardown"]) - args = parser.parse_args() - - logging.basicConfig(level=logging.INFO, format="storage: %(message)s") - - if args.command == "setup": - setup() - elif args.command == "teardown": - teardown() - - -def setup(): - create_dir(BASE_DIR) - - if not os.path.exists(LOOP1): - create_loop_device(LOOP1, BACKING1) - - if not os.path.exists(LOOP2): - create_loop_device(LOOP2, BACKING2) - create_dir(MOUNTPOINT) - create_filesystem(LOOP2, MOUNTPOINT) - - # Sanlock allocates spaces as needed. - with open(FILE, "wb") as f: - f.truncate(0) - - -def teardown(): - if is_mounted(MOUNTPOINT): - remove_filesystem(MOUNTPOINT) - - if os.path.exists(LOOP2): - remove_loop_device(LOOP2, BACKING2) - - if os.path.exists(LOOP1): - remove_loop_device(LOOP1, BACKING1) - - -def create_loop_device(link_path, backing_file, size=1024**3, - sector_size=4096): - log.info("Creating loop device %s", link_path) - - with open(backing_file, "wb") as f: - f.truncate(size) - - out = subprocess.check_output([ - "sudo", - "losetup", - "-f", backing_file, - "--sector-size", str(sector_size), - "--show", - ]) - - device = out.decode("utf-8").strip() - - # Remove stale symlink. - if os.path.islink(link_path): - os.unlink(link_path) - - os.symlink(device, link_path) - chown(link_path) - - -def remove_loop_device(link_path, backing_file): - log.info("Removing loop device %s", link_path) - - subprocess.check_call(["sudo", "losetup", "-d", link_path]) - remove_file(link_path) - remove_file(backing_file) - - -def create_filesystem(device, mountpoint): - log.info("Creating filesystem %s", mountpoint) - - subprocess.check_call(["sudo", "mkfs.xfs", "-q", device]) - subprocess.check_call(["sudo", "mount", device, mountpoint]) - chown(mountpoint) - - -def remove_filesystem(mountpoint): - log.info("Removing filesystem %s", mountpoint) - subprocess.check_call(["sudo", "umount", mountpoint]) - - -def is_mounted(mountpoint): - with open("/proc/self/mounts") as f: - for line in f: - if mountpoint in line: - return True - return False - - -def chown(path): - user_group = "%(USER)s:%(USER)s" % os.environ - subprocess.check_call(["sudo", "chown", user_group, path]) - - -def create_dir(path): - try: - os.makedirs(path) - except EnvironmentError as e: - if e.errno != errno.EEXIST: - raise - - -def remove_file(path): - try: - os.remove(path) - except EnvironmentError as e: - if e.errno != errno.ENOENT: - raise +GiB = 1024**3 +BASE_DIR = "/var/tmp/sanlock-storage" -if __name__ == "__main__": - main() +BACKENDS = { + + "file": + File( + Mount( + LoopDevice( + base_dir=BASE_DIR, + name="file", + size=GiB, + sector_size=4096))), + + "block": + LoopDevice( + base_dir=BASE_DIR, + name="loop", + size=GiB, + sector_size=4096), + +} diff --git a/tox.ini b/tox.ini index 8847b34..c0cf5a1 100644 --- a/tox.ini +++ b/tox.ini @@ -17,6 +17,7 @@ setenv = whitelist_externals = make deps = pytest + userstorage>=0.5.1 commands = py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}" pytest {posargs} -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[sanlock] 02/04: tox.ini: add py310 to environments
by pagure＠pagure.io 27 Nov '22

27 Nov '22

This is an automated email from the git hooks/post-receive script. nsoffer pushed a commit to branch master in repository sanlock. commit 7de76581b25d36375638478fb487ac45ef9925df Author: Albert Esteve <aesteve(a)redhat.com> AuthorDate: Wed Nov 9 14:50:55 2022 +0100 tox.ini: add py310 to environments Add py310 to the list of tox environments, so that we can tests newer versions too. Signed-off-by: Albert Esteve <aesteve(a)redhat.com> --- tox.ini | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tox.ini b/tox.ini index ee871ef..8847b34 100644 --- a/tox.ini +++ b/tox.ini @@ -4,7 +4,7 @@ # and then run "tox" from this directory. [tox] -envlist = py{36,37,38,39},flake8 +envlist = py{36,37,38,39,310},flake8 skipsdist = True skip_missing_interpreters = True @@ -18,7 +18,7 @@ whitelist_externals = make deps = pytest commands = - py{36,37,38}: make BUILDARGS="--build-lib={envsitepackagesdir}" + py{36,37,38,39,310}: make BUILDARGS="--build-lib={envsitepackagesdir}" pytest {posargs} [testenv:flake8] -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[sanlock] 01/04: README.dev: update prerequisites
by pagure＠pagure.io 27 Nov '22

27 Nov '22

This is an automated email from the git hooks/post-receive script. nsoffer pushed a commit to branch master in repository sanlock. commit 0cb0978b7125609443bd69bba2059e68b4b18e14 Author: Albert Esteve <aesteve(a)redhat.com> AuthorDate: Wed Nov 9 14:50:54 2022 +0100 README.dev: update prerequisites Add uuid to the listed prerequisites to avoid the build error due to a missing include: fatal error: uuid/uuid.h: No such file or directory #include <uuid/uuid.h> Signed-off-by: Albert Esteve <aesteve(a)redhat.com> --- README.dev | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.dev b/README.dev index 7f3df2c..1bf1d17 100644 --- a/README.dev +++ b/README.dev @@ -3,11 +3,11 @@ Prerequisites For Fedora/CentOS install the following packages: - $ sudo yum install -y gcc make libaio-devel libblkid-devel + $ sudo yum install -y gcc make libaio-devel libblkid-devel libuuid-devel For Ubuntu/Debian install the following packages: - $ sudo apt install -y gcc make libaio-dev libblkid-dev + $ sudo apt install -y gcc make libaio-dev libblkid-dev uuid-dev How to test sanlock -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

[sanlock] 02/02: wdmd: close watchdog when not used
by pagure＠pagure.io 15 Nov '22

15 Nov '22

This is an automated email from the git hooks/post-receive script. teigland pushed a commit to branch master in repository sanlock. commit 45563a6e0c9d388f58178e03c9dd0b2708b2969c Author: David Teigland <teigland(a)redhat.com> AuthorDate: Mon Nov 14 16:04:39 2022 -0600 wdmd: close watchdog when not used Previously, the watchdog would remain open with keepalives even after all clients (sanlock lockspaces) were removed. This was an unnecessary liability in case it couldn't be kept alive while not needed. It also prevented sanlock from being restarted using a new watchdog timeout. --- wdmd/main.c | 59 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 56 insertions(+), 3 deletions(-) diff --git a/wdmd/main.c b/wdmd/main.c index e89ab5e..3c60b4e 100644 --- a/wdmd/main.c +++ b/wdmd/main.c @@ -101,6 +101,7 @@ struct client { int pid; int pid_dead; int refcount; + int internal; uint64_t renewal; uint64_t expire; void *workfn; @@ -281,12 +282,12 @@ static void dump_debug(int fd) now = monotime(); memset(line, 0, sizeof(line)); - snprintf(line, 255, "wdmd %d socket_gid %d high_priority %d now %llu last_keepalive %llu last_closeunclean %llu allow_scripts %d kill_script_sec %d\n", + snprintf(line, 255, "wdmd %d socket_gid %d high_priority %d now %llu last_keepalive %llu last_closeunclean %llu allow_scripts %d kill_script_sec %d fire_timeout %d\n", getpid(), socket_gid, high_priority, (unsigned long long)now, (unsigned long long)last_keepalive, (unsigned long long)last_closeunclean, - allow_scripts, kill_script_sec); + allow_scripts, kill_script_sec, fire_timeout); line_len = strlen(line); strncat(debug_buf, line, LINE_SIZE); @@ -648,6 +649,7 @@ static int setup_clients(void) ci = client_add(fd, process_listener, client_pid_dead); strncpy(client[ci].name, "listen", WDMD_NAME_SIZE); + client[ci].internal = 1; return 0; } @@ -731,6 +733,25 @@ static int active_clients(void) return 0; } +static void count_clients(int *active, int *external) +{ + int act = 0, ext = 0; + int i; + + for (i = 0; i < client_size; i++) { + if (!client[i].used) + continue; + if (client[i].refcount) + act++; + if (!client[i].internal) + ext++; + } + if (active) + *active = act; + if (external) + *external = ext; +} + #ifdef TEST_FILES #define FILES_DIR "/run/wdmd/test_files" @@ -1447,6 +1468,7 @@ static int setup_signals(void) ci = client_add(fd, process_signals, client_pid_dead); strncpy(client[ci].name, "signal", WDMD_NAME_SIZE); + client[ci].internal = 1; return 0; } @@ -1490,6 +1512,8 @@ static int test_loop(void) void (*workfn) (int ci); void (*deadfn) (int ci); uint64_t test_time; + int resetting = 0; + int active_usage, external_usage; int poll_timeout; int sleep_seconds; int fail_count; @@ -1520,7 +1544,9 @@ static int test_loop(void) } } - if (daemon_quit && !active_clients()) + count_clients(&active_usage, &external_usage); + + if (daemon_quit && !active_usage) break; /* @@ -1529,6 +1555,31 @@ static int test_loop(void) if (!test_loop_enable) continue; + /* + * active_usage are client connections with a refcount. + * external_usage are any clients other than internal. + * (open_watchdog happens with external but not active + * connections.) + * + * checking resetting here is critical to avoiding + * unnecessary resets: while in recovery mode we + * have done close_watchdog_unclean, then all clients + * are cleared, and we need the loop below to see + * no further failures and reopen and pet the watchdog + * again to avoid a reset. After it's been reopened, + * and no longer used due to all clients being cleared, + * then it's ok to get here and close cleanly. + */ + if (!active_usage && !external_usage && !resetting) { + log_debug("close watchdog unused"); + close_watchdog(); + test_loop_enable = 0; + test_interval = standard_test_interval; + poll_timeout = test_interval * 1000; + test_time = 0; + continue; + } + if (monotime() - test_time >= test_interval) { test_time = monotime(); log_debug("test_time %llu", @@ -1549,6 +1600,7 @@ static int test_loop(void) } test_interval = standard_test_interval; + resetting = 0; } else { /* If we can patch the kernel so that close does not generate a ping, then we can skip @@ -1557,6 +1609,7 @@ static int test_loop(void) close_watchdog_unclean(); test_interval = RECOVER_TEST_INTERVAL; + resetting = 1; } } -- To stop receiving notification emails like this one, please contact the administrator of this repository.

1 0

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

sanlock-devel November 2022