[releng] Issue #8330: no f30 silverblue updates: ostree composes started by bodhi are failing
dustymabe reported a new issue against the project: `releng` that you are following: `` * Describe the issue
The ostree composes seem to be failing with an error:
``` 2019-03-12 16:00:50 [ERROR ] [FAIL] Ostree (variant Everything, arch x86_64) failed, but going on anyway. 2019-03-12 16:00:50 [ERROR ] 'ascii' codec can't decode byte 0xc2 in position 6271: ordinal not in range(128) ```
An example failed composes issue for this: https://pagure.io/dusty/failed-composes/issue/1642
* When do you need this? (YYYY/MM/DD)
ASAP so people on Fedora SB 30 can get updates
* When is this no longer needed or useful? (YYYY/MM/DD)
F30 EOL
* If we cannot complete your request, what is the impact?
Users of Fedora SB move to something else (possibly not Fedora). ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: `` The ostree composes weren't failing in the normal pungi runs before release (i.e. not run through bodhi). Mohan pointed out one difference between those setups:
``` 09:55:46* lsedlar | mboddu, is there the same python version as well? 09:56:29 +mboddu | lsedlar: Nope 09:57:06* +mboddu | bodhi-backend01.phx2.fedoraproject.org has python3-3.7.3-1.fc29.x86_64 09:57:19* +mboddu | where as the compose-x86-01.phx2.fedoraproject.org has | python3-3.7.2-5.fc29.x86_64 09:58:01 +mboddu | lsedlar: bodhi-bakend01 has the latest version compared to both of them
``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
walters added a new comment to an issue you are following: `` One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses by default. Something like this?
``` diff --git a/pungi/wrappers/kojiwrapper.py b/pungi/wrappers/kojiwrapper.py index 3cd9ac9..02855bb 100644 --- a/pungi/wrappers/kojiwrapper.py +++ b/pungi/wrappers/kojiwrapper.py @@ -147,7 +147,8 @@ class KojiWrapper(object): task_id = None with self.get_koji_cmd_env() as env: retcode, output = run(command, can_fail=True, logfile=log_file, - show_cmd=True, env=env, universal_newlines=True) + show_cmd=True, env=env, universal_newlines=True, + encoding='UTF-8') if "--task-id" in command: first_line = output.splitlines()[0] if re.match(r'^\d+$', first_line): ``` ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
jlebon added a new comment to an issue you are following: `` So reading the backstory here in https://pagure.io/dusty/failed-composes/issue/1642, it seems like it's getting thrown off by some specific Unicode characters rpm-ostree emits. And some changes in Unicode handling in 3.7.3 is making this reliably fail. Seems like we should be able to just tweak pungi to decode rpm-ostree output using utf-8? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
jlebon added a new comment to an issue you are following: `` In the worst case we *could* tweak rpm-ostree for this, but... it's utf-8. I'm sure rpm-ostree isn't the only app we'd want to run in a runroot that outputs utf-8. :) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
walters added a new comment to an issue you are following: `` This part also looks suspicious:
``` bin/pungi-koji 9:import locale 29:# force C locales 30:locale.setlocale(locale.LC_ALL, "C") ```
I think that should probably be `C.UTF-8` nowadays.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
walters added a new comment to an issue you are following: `` This kobo function (used in a few places in pungi) also looks suspicious:
``` def save_to_file(filename, text, append=False, mode=0o644): """Save text to a file.""" if append and os.path.exists(filename): fd = os.open(filename, os.O_RDWR | os.O_APPEND, mode) else: fd = os.open(filename, os.O_WRONLY | os.O_CREAT | os.O_TRUNC, mode) if not isinstance(text, bytes): text = bytes(text, encoding=locale.getpreferredencoding()) os.write(fd, text) os.close(fd) ```
Having one function to create both binary data and text seems odd...given the use cases I'd just assume any `bytes()` instances are `UTF-8`?
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
walters added a new comment to an issue you are following: `` This bug is pretty urgent IMO because we shipped people an operating system and aren't giving them security updates - anything I can do to help please let me know. If truly necessary we can try to strip out the UTF-8 usage in rpm-ostree but...can a Pungi maintainer comment on https://pagure.io/releng/issue/8330#comment-569167 ? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
mohanboddu added a new comment to an issue you are following: `` I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and started f30-updates and f30-updates-testing push. Lets see what happens. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: ``
I downgraded bodhi-backend01.phx2.fp.o to python3-3.7.2.-5, and started f30-updates and f30-updates-testing push. Lets see what happens.
Thanks @mohanboddu.
He did this so we can see if we can get a new update out today. The pungi maintainers would only be able to get back to us at the earliest tomorrow morning and we'd like to get an update out sooner than later. Since the attempt at a workaround here is a downgrade this is only a temporary bandaid so we'll have to dig in and possibly implement the suggestions from @walters and @jlebon soon. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
mohanboddu added a new comment to an issue you are following: `` Okay, I am still seeing this after downgrading python3 in the [latest bodhi compose](https://kojipkgs.fedoraproject.org/compose/updates/Fedora-30-updates-2019050...) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
jlebon added a new comment to an issue you are following: `` OK, testing a temporary hack in https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/32. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: ``
OK, testing a temporary hack in https://src.fedoraproject.org/rpms/rpm-ostree/pull-request/32.
cool - once we get a build would we need to use a buildroot override? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
jlebon added a new comment to an issue you are following: `` OK, new rpm-ostree in buildroot with the patch above. Can someone retry the updates & updates-testing push?
/cc @kevin ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
kevin added a new comment to an issue you are following: `` f30 updates-testing away. (if that works we can push updates too, but best to just test with updates-testing)
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
kevin added a new comment to an issue you are following: `` It seems to have worked. If we get a f30-updates I think we can be happy... and look for longer term fix in pungi. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
sinnykumari added a new comment to an issue you are following: `` [f30-updates](https://kojipkgs.fedoraproject.org/compose/updates/Fedora-30-updates-2019050...) succeeded as well. Thanks all!
Both updates and testing ref has not yet synced to production ostree/repo/refs/ ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: ``
@walters One thing I'm a bit confused on is why Pungi isn't doing UTF-8 for subprocesses by default.
open PR in https://pagure.io/pungi/pull-request/1185
@sinnykumari Both updates and testing ref has not yet synced to production ostree/repo/refs/
right - we'll need to look into getting those synced out and also alias the stable ref to the updates ref so that they are one in the same ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: `` ok - we have an updates ref now - i'm running a quick update on my laptop to make sure it's not obviously broken before I alias the stable ref to it. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
dustymabe added a new comment to an issue you are following: `` it looked good. I got the stable ref aliased over and everyone should be able to update now 🌈 ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are failing ` of project: `releng` has been updated to: Closed as Fixed by dustymabe.
dustymabe added a new comment to an issue you are following: `` actually i'll re-open this as we applied a bandaid (patch to rpm-ostree) but the real fix is in pungi. I'll re-open until we confirm it's fixed (open PR is [here](https://pagure.io/pungi/pull-request/1185)) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are failing ` of project: `releng` has been updated to: Open by dustymabe.
The status of the issue: `no f30 silverblue updates: ostree composes started by bodhi are failing ` of project: `releng` has been updated to: Closed as Fixed by dustymabe.
dustymabe added a new comment to an issue you are following: `` we can close this now since it's been a while ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/8330
rel-eng@lists.fedoraproject.org
-
Colin Walters -
Dusty Mabe
-
Jonathan Lebon
-
Kevin Fenzi
-
Mohan Boddu
-
Sinny Kumari