Hello,
We ran automated tests on a recent commit from this kernel tree:
Kernel repo: https://gitlab.com/cki-project/kernel-ark.git Commit: 889afeef8da3 - [redhat] kernel-5.10.0-0.rc0.20201015gitb5fc7a89e58b.41
The results of these automated tests are provided below.
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
All kernel binaries, config files, and logs are available for download here:
https://arr-cki-prod-datawarehouse-public.s3.amazonaws.com/index.html?prefix...
We attempted to compile the kernel for multiple architectures, but the compile failed on one or more architectures:
s390x: FAILED (see build-s390x.log.xz attachment)
We hope that these logs can help you find the problem quickly. For the full detail on our testing procedures, please scroll to the bottom of this message.
Please reply to this email if you have any questions about the tests that we ran or if you have any suggestions on how to make future tests more effective.
,-. ,-. ( C ) ( K ) Continuous `-',-.`-' Kernel ( I ) Integration `-' ______________________________________________________________________________
Compile testing ---------------
We compiled the kernel for 4 architectures:
aarch64: rpmbuild arguments: rpmbuild --target aarch64 --with=cross --with=kabidw_base --without=bpftool --without=selftests --without=debug --without=ipaclones --without=perf --without=tools
ppc64le: rpmbuild arguments: rpmbuild --target ppc64le --with=cross --with=kabidw_base --without=bpftool --without=selftests --without=debug --without=ipaclones --without=perf --without=tools
s390x: rpmbuild arguments: rpmbuild --target s390x --with=cross --with=kabidw_base --without=bpftool --without=selftests --without=debug --without=ipaclones --without=perf --without=tools
x86_64: rpmbuild arguments: rpmbuild --target x86_64 --with=kabidw_base --without=debug
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
Or is there anyone on the kernel@ list that finds them useful?
Thanks,
Jiri
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Or is there anyone on the kernel@ list that finds them useful?
It seems nobody does.
My take is those emails should be switched off. CKI team, could you please do so?
Thanks,
Jiri
----- Original Message -----
From: "Jiri Benc" jbenc@redhat.com To: "CKI Project" cki-project@redhat.com Cc: "Don Zickus" dzickus@redhat.com, kernel@lists.fedoraproject.org, "Veronika Kabatova" vkabatov@redhat.com Sent: Monday, October 19, 2020 11:09:57 AM Subject: Re: ❌ FAIL: Test report for kernel 5.9.0-42.test.fc33 (ark)
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Hi,
where the results go is ARK maintainers' decision and not ours. Jeremy and Justin picked this list for the results. If you believe a different list should be used, please discuss this with the maintainers and we can change the list based on their decision.
FWIW, most of the current failures are related to broken ARK process (which Don started working on fixing recently) and different unset configs, so the emails should be actionable for people who care about those things -- like the maintainers. If those failures are not relevant to you that's totally fine and in that case (especially if the maintainers decide to keep this list for the results), setting up a filter is the right thing to do.
Veronika
Or is there anyone on the kernel@ list that finds them useful?
It seems nobody does.
My take is those emails should be switched off. CKI team, could you please do so?
Thanks,
Jiri _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
On Mon, Oct 19, 2020 at 6:23 AM Veronika Kabatova vkabatov@redhat.com wrote:
----- Original Message -----
From: "Jiri Benc" jbenc@redhat.com To: "CKI Project" cki-project@redhat.com Cc: "Don Zickus" dzickus@redhat.com, kernel@lists.fedoraproject.org, "Veronika Kabatova" vkabatov@redhat.com Sent: Monday, October 19, 2020 11:09:57 AM Subject: Re: ❌ FAIL: Test report for kernel 5.9.0-42.test.fc33 (ark)
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Hi,
where the results go is ARK maintainers' decision and not ours. Jeremy and Justin picked this list for the results. If you believe a different list should be used, please discuss this with the maintainers and we can change the list based on their decision.
FWIW, most of the current failures are related to broken ARK process (which Don started working on fixing recently) and different unset configs, so the emails should be actionable for people who care about those things -- like the maintainers. If those failures are not relevant to you that's totally fine and in that case (especially if the maintainers decide to keep this list for the results), setting up a filter is the right thing to do.
Veronika
Or is there anyone on the kernel@ list that finds them useful?
It seems nobody does.
My take is those emails should be switched off. CKI team, could you please do so?
I find the emails useful. If we don't want them on the list for some reason, that is fine, just redirect them to kernel-team@fedoraproject.org which for now is just me, but would go to any current Fedora kernel maintainers.
Justin
On Mon, Oct 19, 2020 at 11:09:57AM +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Or is there anyone on the kernel@ list that finds them useful?
It seems nobody does.
My take is those emails should be switched off. CKI team, could you please do so?
I had thought this was fixed now with merges from two weeks ago. Let me check things.
Cheers, Don
On Mon, Oct 19, 2020 at 11:09:57AM +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
00:01:13 BUILDING A KERNEL FOR s390x... 00:01:13 ++ perl -e 'print "42.test.fc34" =~ s/.pr.[0-9A-Fa-f]{32}//r' 00:01:13 + ShortRel=42.test.fc34 00:01:13 + perl -p -i -e 's/^EXTRAVERSION.*/EXTRAVERSION = -42.test.fc34.s390x/' Makefile 00:01:13 + perl -p -i -e 's/^PATCHLEVEL.*/PATCHLEVEL = 9/' Makefile 00:01:13 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= -j30 mrproper 00:01:16 + cp configs/kernel-5.9.0-s390x.config .config 00:01:16 + cp /tmp/rpmbuild/SOURCES/x509.genkey certs/. 00:01:16 ++ head -1 .config 00:01:16 ++ cut -b 3- 00:01:16 + Arch=s390 00:01:16 + echo USING ARCH=s390 00:01:16 USING ARCH=s390 00:01:16 + KCFLAGS= 00:01:16 + '[' '' == '' ']' 00:01:16 + KCFLAGS=' ' 00:01:16 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 olddefconfig 00:01:18 + perl -p -i -e 's/^CONFIG_BUILD_SALT.*/CONFIG_BUILD_SALT="5.9.0-42.test.fc34.s390x"/' .config 00:01:18 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 'KCFLAGS= ' WITH_GCOV=0 -j30 bzImage 00:01:23 00:01:23 Auto-detecting system features: 00:01:23 ... libelf: [ [32mon[m ] 00:01:23 ... zlib: [ [32mon[m ] 00:01:23 ... bpf: [ [32mon[m ] 00:01:23 00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported 00:04:12 FAILED: load BTF from vmlinux: Unknown error -4001make: *** [Makefile:1166: vmlinux] Error 255 00:04:12 error: Bad exit status from /var/tmp/rpm-tmp.dfy0KE (%build)
Cheers, Don
On Mon, Oct 19, 2020 at 3:12 PM Don Zickus dzickus@redhat.com wrote:
On Mon, Oct 19, 2020 at 11:09:57AM +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
But the Fedora kernel-tools package builds just fine against ELN, including libbpf on s390x. https://koji.fedoraproject.org/koji/taskinfo?taskID=53787549
Justin
00:01:13 BUILDING A KERNEL FOR s390x... 00:01:13 ++ perl -e 'print "42.test.fc34" =~ s/.pr.[0-9A-Fa-f]{32}//r' 00:01:13 + ShortRel=42.test.fc34 00:01:13 + perl -p -i -e 's/^EXTRAVERSION.*/EXTRAVERSION = -42.test.fc34.s390x/' Makefile 00:01:13 + perl -p -i -e 's/^PATCHLEVEL.*/PATCHLEVEL = 9/' Makefile 00:01:13 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= -j30 mrproper 00:01:16 + cp configs/kernel-5.9.0-s390x.config .config 00:01:16 + cp /tmp/rpmbuild/SOURCES/x509.genkey certs/. 00:01:16 ++ head -1 .config 00:01:16 ++ cut -b 3- 00:01:16 + Arch=s390 00:01:16 + echo USING ARCH=s390 00:01:16 USING ARCH=s390 00:01:16 + KCFLAGS= 00:01:16 + '[' '' == '' ']' 00:01:16 + KCFLAGS=' ' 00:01:16 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 olddefconfig 00:01:18 + perl -p -i -e 's/^CONFIG_BUILD_SALT.*/CONFIG_BUILD_SALT="5.9.0-42.test.fc34.s390x"/' .config 00:01:18 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 'KCFLAGS= ' WITH_GCOV=0 -j30 bzImage 00:01:23 00:01:23 Auto-detecting system features: 00:01:23 ... libelf: [ [32mon[m ] 00:01:23 ... zlib: [ [32mon[m ] 00:01:23 ... bpf: [ [32mon[m ] 00:01:23 00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported 00:04:12 FAILED: load BTF from vmlinux: Unknown error -4001make: *** [Makefile:1166: vmlinux] Error 255 00:04:12 error: Bad exit status from /var/tmp/rpm-tmp.dfy0KE (%build)
Cheers, Don _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
On Mon, 19 Oct 2020 16:12:12 -0400, Don Zickus wrote:
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
[...]
00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported
Thanks for the log. The problem seems to be the cross-compiling.
While eBPF was supposed to be architecture independent byte code, the authors failed in reaching the goal - eBPF is not endianess neutral. eBPF byte code compiled on a little endian machine cannot be used on a big endian machine and vice versa.
Now, libbpf does non-trivial rewriting of eBPF code (you can think of it as a compiler of sort). And apparently, libbpf compiled and running on a little endian machine cannot handle big endian eBPF byte code.
I'm not sure what exactly is causing this failure. My guess would be CONFIG_DEBUG_INFO_BTF. If I'm right, we'll probably have to disable it on s390x while cross compiling for the time being. Could you try a cross-compile with CONFIG_DEBUG_INFO_BTF disabled? It's something that will have to be done only for cross-compiling, we want that config option enabled on native builds.
On a related note, if this is the cause of the CKI failures, I wonder why the log snippet you quoted was not the very first thing in the emails? We talked about the CKI email format before and I think there's still a lot of room for improvements there. It should be immediately obvious what went wrong from the email, while currently, it requires a CKI expert to actually interpret it.
Thanks,
Jiri
----- Original Message -----
From: "Jiri Benc" jbenc@redhat.com To: "Don Zickus" dzickus@redhat.com Cc: "CKI Project" cki-project@redhat.com, kernel@lists.fedoraproject.org, "Veronika Kabatova" vkabatov@redhat.com Sent: Tuesday, October 20, 2020 9:45:54 AM Subject: Re: ❌ FAIL: Test report for kernel 5.9.0-42.test.fc33 (ark)
On Mon, 19 Oct 2020 16:12:12 -0400, Don Zickus wrote:
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
[...]
00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported
These are older pipelines that are waiting for
https://lore.kernel.org/bpf/20201008234000.740660-9-andrii@kernel.org/T/#u
to be merged into mainline/ARK. We're also in contact with Jirka Olsa due to a different bpf issue with resolve_btfids (not s390x-specific).
If you check out the new results, they are due to broken ARK configs.
Thanks for the log. The problem seems to be the cross-compiling.
While eBPF was supposed to be architecture independent byte code, the authors failed in reaching the goal - eBPF is not endianess neutral. eBPF byte code compiled on a little endian machine cannot be used on a big endian machine and vice versa.
Now, libbpf does non-trivial rewriting of eBPF code (you can think of it as a compiler of sort). And apparently, libbpf compiled and running on a little endian machine cannot handle big endian eBPF byte code.
I'm not sure what exactly is causing this failure. My guess would be CONFIG_DEBUG_INFO_BTF. If I'm right, we'll probably have to disable it on s390x while cross compiling for the time being. Could you try a cross-compile with CONFIG_DEBUG_INFO_BTF disabled? It's something that will have to be done only for cross-compiling, we want that config option enabled on native builds.
On a related note, if this is the cause of the CKI failures, I wonder why the log snippet you quoted was not the very first thing in the emails? We talked about the CKI email format before and I think there's still a lot of room for improvements there. It should be immediately obvious what went wrong from the email, while currently, it requires a CKI expert to actually interpret it.
Agreed. We do have this functionality on the list. Unfortunately the implementation was pushed back due to all the work on the workflow changes, as we don't have time/people to work on everything at once.
As I side note, I'll be updating the email configs based on Justin's suggestion later today.
Veronika
Thanks,
Jiri _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
On Tue, Oct 20, 2020 at 04:37:16AM -0400, Veronika Kabatova wrote:
----- Original Message -----
From: "Jiri Benc" jbenc@redhat.com To: "Don Zickus" dzickus@redhat.com Cc: "CKI Project" cki-project@redhat.com, kernel@lists.fedoraproject.org, "Veronika Kabatova" vkabatov@redhat.com Sent: Tuesday, October 20, 2020 9:45:54 AM Subject: Re: ❌ FAIL: Test report for kernel 5.9.0-42.test.fc33 (ark)
On Mon, 19 Oct 2020 16:12:12 -0400, Don Zickus wrote:
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
[...]
00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported
These are older pipelines that are waiting for
https://lore.kernel.org/bpf/20201008234000.740660-9-andrii@kernel.org/T/#u
to be merged into mainline/ARK. We're also in contact with Jirka Olsa due to a different bpf issue with resolve_btfids (not s390x-specific).
also probably cross build issue with following error:
HOSTCC /usr/src/kernels/5.9.0/tools/bpf/resolve_btfids/fixdep.o /bin/sh: /usr/src/kernels/5.9.0/tools/bpf/resolve_btfids//fixdep: cannot execute binary file: Exec format error
which I was hoping was fixed by: dc3652d3f0d5 tools resolve_btfids: Always force HOSTARCH 11bb2f7a4590 bpf: Fix cross build for CONFIG_DEBUG_INFO_BTF option
but might not be the case, so Veronika is preparing reproducer, because I can't recreate this one
thanks, jirka
On Mon, Oct 19, 2020 at 04:12:12PM -0400, Don Zickus wrote:
On Mon, Oct 19, 2020 at 11:09:57AM +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 12:20:41 +0200, Jiri Benc wrote:
On Thu, 15 Oct 2020 09:13:27 -0000, CKI Project wrote:
Overall result: FAILED (see details below) Merge: OK Compile: FAILED
Given the failure emails are sent to kernel@lists.fedoraproject.org daily, I wonder, is there any value in them? I'm just ignoring them and considering setting up a filter, as they are not actionable for anyone and thus are just noise.
CKI team, could you please comment on this?
Sorry about that. I thought things were failing because of ARK infra but now they are failing due to libbpf on s390x. Does bpf compile there?
00:01:13 BUILDING A KERNEL FOR s390x... 00:01:13 ++ perl -e 'print "42.test.fc34" =~ s/.pr.[0-9A-Fa-f]{32}//r' 00:01:13 + ShortRel=42.test.fc34 00:01:13 + perl -p -i -e 's/^EXTRAVERSION.*/EXTRAVERSION = -42.test.fc34.s390x/' Makefile 00:01:13 + perl -p -i -e 's/^PATCHLEVEL.*/PATCHLEVEL = 9/' Makefile 00:01:13 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= -j30 mrproper 00:01:16 + cp configs/kernel-5.9.0-s390x.config .config 00:01:16 + cp /tmp/rpmbuild/SOURCES/x509.genkey certs/. 00:01:16 ++ head -1 .config 00:01:16 ++ cut -b 3- 00:01:16 + Arch=s390 00:01:16 + echo USING ARCH=s390 00:01:16 USING ARCH=s390 00:01:16 + KCFLAGS= 00:01:16 + '[' '' == '' ']' 00:01:16 + KCFLAGS=' ' 00:01:16 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 olddefconfig 00:01:18 + perl -p -i -e 's/^CONFIG_BUILD_SALT.*/CONFIG_BUILD_SALT="5.9.0-42.test.fc34.s390x"/' .config 00:01:18 + /usr/bin/make CROSS_COMPILE=s390x-linux-gnu- -s HOSTCFLAGS= HOSTLDFLAGS= ARCH=s390 'KCFLAGS= ' WITH_GCOV=0 -j30 bzImage 00:01:23 00:01:23 Auto-detecting system features: 00:01:23 ... libelf: [ [32mon[m ] 00:01:23 ... zlib: [ [32mon[m ] 00:01:23 ... bpf: [ [32mon[m ] 00:01:23 00:01:25 LINK resolve_btfids 00:04:11 libbpf: non-native ELF endianness is not supported
there was recent change for libbpf that should fix this: 3289959b97ca libbpf: Support BTF loading and raw data output in both endianness
jirka
00:04:12 FAILED: load BTF from vmlinux: Unknown error -4001make: *** [Makefile:1166: vmlinux] Error 255 00:04:12 error: Bad exit status from /var/tmp/rpm-tmp.dfy0KE (%build)
Cheers, Don _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org
kernel@lists.fedoraproject.org