On Mon, 2024-04-01 at 08:53 -0700, Adam Williamson wrote:
Hi folks! I just discovered this so I'm still investigating it,
but
wanted to give a quick heads-up.
It looks like the message consumers on openqa01 all broke on Saturday
when a fedora-messaging update landed. This affects a lot of things,
but by far the most important is that openQA test results are not
getting reported to resultsdb. This will be causing all critpath
updates to be stuck in gating.
I am going to investigate this urgently, of course, and as a stopgap I
will manually trigger submission of all reports from the last couple of
days shortly. Very sorry for the inconvenience.
Also affected:
https://openqa.fedoraproject.org/nightlies.html and
.json are not getting updated, validation test events are not being
created, check-compose emails not being generated, possibly something
else I've forgotten.
OK, I've found the cause of this: a 'modernization' of the fedora-
messaging spec caused the hashbang of /usr/bin/fedora-messaging to
change so it is run with `python3 -sP`, which causes python not to use
libraries from /usr/local/lib . This was a surprising and unexpected
change in a stable release update, and it may affect other folks too.
See
https://bugzilla.redhat.com/show_bug.cgi?id=2272526 .
I'll work around this for now and wait to see what comes of the bug
report. The consumers will gradually catch up with all the stuff they
should have been doing for the last two days over the next little
while.
--
Adam Williamson (he/him/his)
Fedora QA
Fedora Chat: @adamwill:fedora.im | Mastodon: @adamw(a)fosstodon.org
https://www.happyassassin.net