Have we ever seen this before? Comes today's rawhide compose [1]
DEBUG util.py:439: Committing: 100% DEBUG util.py:439: error: unlinkat: No such file or directory DEBUG util.py:439: Traceback (most recent call last): DEBUG util.py:439: File "/usr/bin/pungi-make-ostree", line 15, in <module> DEBUG util.py:439: ostree.main() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/__init__.py", line 89, in main DEBUG util.py:439: func() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 99, in run DEBUG util.py:439: self._make_tree() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 46, in _make_tree DEBUG util.py:439: shortcuts.run(cmd, show_cmd=True, stdout=True, logfile=log_file) DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/kobo/shortcuts.py", line 335, in run DEBUG util.py:439: raise RuntimeError(err_msg) DEBUG util.py:439: RuntimeError: ERROR running command: rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/rawhide/ --write-commitid-to=/mnt/koji/compose/rawhide/Fedora-Rawhide-20170622.n.0/logs/x86_64/Atomic/ostree-3/commitid.log --add-metadata-string=version=Rawhide.20170622.n.0 /mnt/koji/compose/rawhide/Fedora-Rawhide-20170622.n.0/work/ostree-3/config_repo/fedora-atomic-docker-host.json DEBUG util.py:577: Child return code was: 1 DEBUG util.py:188: kill orphans
[1] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20101234
happened again last night: https://kojipkgs.fedoraproject.org//work/tasks/7733/20127733/root.log
On 06/22/2017 09:40 PM, Dusty Mabe wrote:
Have we ever seen this before? Comes today's rawhide compose [1]
DEBUG util.py:439: Committing: 100% DEBUG util.py:439: error: unlinkat: No such file or directory DEBUG util.py:439: Traceback (most recent call last): DEBUG util.py:439: File "/usr/bin/pungi-make-ostree", line 15, in <module> DEBUG util.py:439: ostree.main() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/__init__.py", line 89, in main DEBUG util.py:439: func() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 99, in run DEBUG util.py:439: self._make_tree() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 46, in _make_tree DEBUG util.py:439: shortcuts.run(cmd, show_cmd=True, stdout=True, logfile=log_file) DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/kobo/shortcuts.py", line 335, in run DEBUG util.py:439: raise RuntimeError(err_msg) DEBUG util.py:439: RuntimeError: ERROR running command: rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/rawhide/ --write-commitid-to=/mnt/koji/compose/rawhide/Fedora-Rawhide-20170622.n.0/logs/x86_64/Atomic/ostree-3/commitid.log --add-metadata-string=version=Rawhide.20170622.n.0 /mnt/koji/compose/rawhide/Fedora-Rawhide-20170622.n.0/work/ostree-3/config_repo/fedora-atomic-docker-host.json DEBUG util.py:577: Child return code was: 1 DEBUG util.py:188: kill orphans
[1] - https://koji.fedoraproject.org/koji/taskinfo?taskID=20101234
On Sat, Jun 24, 2017, at 09:34 AM, Dusty Mabe wrote:
happened again last night: https://kojipkgs.fedoraproject.org//work/tasks/7733/20127733/root.log
We landed some PRs upstream to help debug this, but they were hard to backport. I just manually started a build of a git snapshot: https://koji.fedoraproject.org/koji/taskinfo?taskID=20144391
Though the most useful would be someone on the server capturing an strace.
It does look like something went badly wrong though here: https://kojipkgs.fedoraproject.org/compose/atomic/
That directory looks like it was initialized as a repo, when it was formerly just a directory with multiple repos. It looks empty https://kojipkgs.fedoraproject.org/compose/atomic/refs/heads/ (Though of course my strong preference would be for us to have a single ostree repo for FAH, but let's not scope that into this)
A configuration failure somewhere? Presumably either pungi or bodhi?
On 06/24/2017 10:41 AM, Colin Walters wrote:
On Sat, Jun 24, 2017, at 09:34 AM, Dusty Mabe wrote:
happened again last night: https://kojipkgs.fedoraproject.org//work/tasks/7733/20127733/root.log
We landed some PRs upstream to help debug this, but they were hard to backport. I just manually started a build of a git snapshot: https://koji.fedoraproject.org/koji/taskinfo?taskID=20144391
cool. I'll keep an eye on it to see if I see it again.
Though the most useful would be someone on the server capturing an strace.
It does look like something went badly wrong though here: https://kojipkgs.fedoraproject.org/compose/atomic/
That directory looks like it was initialized as a repo, when it was formerly just a directory with multiple repos. It looks empty https://kojipkgs.fedoraproject.org/compose/atomic/refs/heads/ (Though of course my strong preference would be for us to have a single ostree repo for FAH, but let's not scope that into this)
A configuration failure somewhere? Presumably either pungi or bodhi?
Yeah. looks like it happened in 2016, though. We probably just need to have it cleaned up, right?
Carrying this forward to the atomic-devel mailing list from the following fedora cloud list discussion:
https://lists.fedoraproject.org/archives/list/cloud@lists.fedoraproject.org/...
On 06/25/2017 09:26 PM, Dusty Mabe wrote:
On 06/24/2017 10:41 AM, Colin Walters wrote:
On Sat, Jun 24, 2017, at 09:34 AM, Dusty Mabe wrote:
happened again last night: https://kojipkgs.fedoraproject.org//work/tasks/7733/20127733/root.log
We landed some PRs upstream to help debug this, but they were hard to backport. I just manually started a build of a git snapshot: https://koji.fedoraproject.org/koji/taskinfo?taskID=20144391
cool. I'll keep an eye on it to see if I see it again.
I see this from Saturday's F27 ppc64le run:
DEBUG util.py:439: Committing: 100% DEBUG util.py:439: error: Commit: unlinkat: Directory not empty DEBUG util.py:439: Traceback (most recent call last): DEBUG util.py:439: File "/usr/bin/pungi-make-ostree", line 15, in <module> DEBUG util.py:439: ostree.main() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/__init__.py", line 89, in main DEBUG util.py:439: func() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 90, in run DEBUG util.py:439: self._make_tree() DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/pungi/ostree/tree.py", line 38, in _make_tree DEBUG util.py:439: shortcuts.run(cmd, show_cmd=True, stdout=True, logfile=log_file) DEBUG util.py:439: File "/usr/lib/python2.7/site-packages/kobo/shortcuts.py", line 335, in run DEBUG util.py:439: raise RuntimeError(err_msg) DEBUG util.py:439: RuntimeError: ERROR running command: rpm-ostree compose tree --repo=/mnt/koji/compose/atomic/27/ --write-commitid-to=/mnt/koji/compose/branched/Fedora-27-20170902.n.0/logs/ppc64le/Atomic/ostree-2/commitid.log --add-metadata-string=version=27.20170902.n.0 /mnt/koji/compose/branched/Fedora-27-20170902.n.0/work/ostree-2/config_repo/fedora-atomic-host.json DEBUG util.py:577: Child return code was: 1
https://koji.fedoraproject.org/koji/taskinfo?taskID=21618968
-
Colin Walters -
Dusty Mabe