fedora-ci.koji-build.tier0.functional fails due to "ERROR: Test environment installation failed: reason unknown, please escalate"
There's an update in Bodhi that's stuck right now because fedora- ci.koji-build.tier0.functional failed for lorax, which has it configured as a blocking test in its gating.yaml. This is the test:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipel...
the console output shows:
Waiting for Testing Farm... [Pipeline] waitForWebhook [Pipeline] retry [Pipeline] { [Pipeline] httpRequest [Pipeline] } [Pipeline] // retry [Pipeline] echo The status is now "running" [Pipeline] echo Waiting for Testing Farm... [Pipeline] waitForWebhook [Pipeline] retry [Pipeline] { [Pipeline] httpRequest [Pipeline] } [Pipeline] // retry [Pipeline] echo The status is now "error" [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Process Test Results (XUnit)) Stage "Process Test Results (XUnit)" skipped due to when conditional [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] catchError [Pipeline] { [Pipeline] error [Pipeline] } ERROR: Test environment installation failed: reason unknown, please escalate
Does anyone know what's going on there? I did retry the test, and it failed the same way both times, so it doesn't seem like a blip.
Hi Adam,
More information is in Testing Farm artifacts, you can find the link in Jenkins log:
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Seems like some conflict:
Error: Problem: package pipewire-plugin-jack-0.3.43-1.fc35.x86_64 requires jack-audio-connection-kit, but none of the providers can be installed - package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64 conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.x86_64 - package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64 conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.i686 - conflicting requests
HTH, /M
On Tue, Jan 11, 2022 at 8:18 PM Adam Williamson adamwill@fedoraproject.org wrote:
There's an update in Bodhi that's stuck right now because fedora- ci.koji-build.tier0.functional failed for lorax, which has it configured as a blocking test in its gating.yaml. This is the test:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipel...
the console output shows:
Waiting for Testing Farm... [Pipeline] waitForWebhook [Pipeline] retry [Pipeline] { [Pipeline] httpRequest [Pipeline] } [Pipeline] // retry [Pipeline] echo The status is now "running" [Pipeline] echo Waiting for Testing Farm... [Pipeline] waitForWebhook [Pipeline] retry [Pipeline] { [Pipeline] httpRequest [Pipeline] } [Pipeline] // retry [Pipeline] echo The status is now "error" [Pipeline] } [Pipeline] // script [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Process Test Results (XUnit)) Stage "Process Test Results (XUnit)" skipped due to when conditional [Pipeline] } [Pipeline] // stage [Pipeline] stage [Pipeline] { (Declarative: Post Actions) [Pipeline] catchError [Pipeline] { [Pipeline] error [Pipeline] } ERROR: Test environment installation failed: reason unknown, please escalate
Does anyone know what's going on there? I did retry the test, and it failed the same way both times, so it doesn't seem like a blip. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, 2022-01-11 at 21:15 +0100, Miroslav Vadkerti wrote:
Hi Adam,
More information is in Testing Farm artifacts, you can find the link in Jenkins log:
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Seems like some conflict:
Error: Problem: package pipewire-plugin-jack-0.3.43-1.fc35.x86_64 requires jack-audio-connection-kit, but none of the providers can be installed
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.x86_64
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.i686
- conflicting requests
Ah, OK, thanks, I see. So that's just this test not really coping with an update containing packages that conflict with default-installed packages, I guess...it's trying to assert that it can install the packages from the update, but in this case it's correct that it can't install pipewire-j-a-c-k since regular j-a-c-k is already installed.
On Tue, Jan 11, 2022 at 10:30 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2022-01-11 at 21:15 +0100, Miroslav Vadkerti wrote:
Hi Adam,
More information is in Testing Farm artifacts, you can find the link in Jenkins log:
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Seems like some conflict:
Error: Problem: package pipewire-plugin-jack-0.3.43-1.fc35.x86_64 requires jack-audio-connection-kit, but none of the providers can be installed
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.x86_64
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.i686
- conflicting requests
Ah, OK, thanks, I see. So that's just this test not really coping with an update containing packages that conflict with default-installed packages, I guess...it's trying to assert that it can install the packages from the update, but in this case it's correct that it can't install pipewire-j-a-c-k since regular j-a-c-k is already installed.
The user has to cope with conflicting packages himself, we are able to exclude some packages from updating using prepare's exclude field, something we use for example in curl:
https://src.fedoraproject.org/rpms/curl/pull-request/11#_2__6
Which has conflicting packages. Maybe that would help here also, maybe not, not sure ...
If there is a way how CI should better handle the update here, we will be happy to fix it.
/M
-- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, 2022-01-12 at 11:47 +0100, Miroslav Vadkerti wrote:
On Tue, Jan 11, 2022 at 10:30 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2022-01-11 at 21:15 +0100, Miroslav Vadkerti wrote:
Hi Adam,
More information is in Testing Farm artifacts, you can find the link in Jenkins log:
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Seems like some conflict:
Error: Problem: package pipewire-plugin-jack-0.3.43-1.fc35.x86_64 requires jack-audio-connection-kit, but none of the providers can be installed
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.x86_64
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.i686
- conflicting requests
Ah, OK, thanks, I see. So that's just this test not really coping with an update containing packages that conflict with default-installed packages, I guess...it's trying to assert that it can install the packages from the update, but in this case it's correct that it can't install pipewire-j-a-c-k since regular j-a-c-k is already installed.
The user has to cope with conflicting packages himself, we are able to exclude some packages from updating using prepare's exclude field, something we use for example in curl:
https://src.fedoraproject.org/rpms/curl/pull-request/11#_2__6
Which has conflicting packages. Maybe that would help here also, maybe not, not sure ...
If there is a way how CI should better handle the update here, we will be happy to fix it.
Well, I guess the interesting wrinkle here is exactly how the test operated. The update in question contained two packages, pipewire and lorax:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-4cf1bb6eed
of those two packages, it's lorax that lists this test as gating in its config file:
https://src.fedoraproject.org/rpms/lorax/blob/rawhide/f/gating.yaml#_7
pipewire does not. And the test reports itself as having run on the lorax package, not on the update. In resultsdb the 'type' is 'koji_build' (i.e. a single Koji build, not a Bodhi update) and the 'item' is 'lorax-35.9-2.fc35'. In CI messages, the 'artifact' dict is all about the 'koji-build' 'lorax-35.9-2.fc35':
https://apps.fedoraproject.org/datagrepper/id?id=2022-eaec10eb-2ecb-4055-8e6...
however, it seems this specific test actually runs as a test of the entire update. It pulls in all the packages from the update and fails if it can't install any of them.
So...we're reporting a failure of a test of "the lorax package" which didn't actually have anything to do with the lorax package. And we're using the gating configuration of the lorax package to decide whether to gate on a test which actually runs against *every package in the update*. From pipewire's perspective, there's no way it can communicate "this test isn't safe for my package, don't run/gate on it" that would work if some *other* package in the update specifies that test as a gating test...
Given the above, I'm not sure we can really say it's ever entirely safe for a package to gate on this test. Because this test is going to function differently depending on what *other* packages are in the same update. Even if a package can affirm that *its own subpackages* should always be installable in any update it's a part of, it can never safely affirm that *all subpackages of any other package in the same update* should always be installable.
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
WDYT?
On Wed, 2022-01-12 at 09:13 -0800, Adam Williamson wrote:
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
Additional: so after writing out this email I thought, hey, I'll go find the source for this test and see if I can fix it.
Whew, boy.
First I went to the CI docs. These do describe the generic tests:
https://docs.fedoraproject.org/en-US/ci/generic_tests/
but provide no information about where the tests actually *come from* (so where you could go to modify them).
Next I checked my Firefox history and remembered there's a Pagure "fedora-ci" team, so I checked the projects under that, but found nothing relevant:
https://pagure.io/group/fedora-ci
so I resorted to web searching. On a search for "fedora rpminspect" (I picked rpminspect as it's a fairly 'unique' name of one of the generic tests) the sixth result was https://github.com/fedora-ci/rpminspect-pipeline , which seems to be the actual implementation of the rpminspect test pipeline. So there's a github fedora-ci as well as the Pagure fedora-ci? Okay. I checked that group's list of projects and found the installability pipeline:
https://github.com/fedora-ci/installability-pipeline
but my search didn't end there, because the test code isn't in the pipeline repo, it pulls it from elsewhere. I had to figure that out. I found it in the 'prepare.sh' script, where it adds a COPR repo and installs a package:
https://github.com/fedora-ci/installability-pipeline/blob/master/prepare.sh#...
so that told me the actual test code was coming from this 'mini-tps' package in this COPR repo. So now I had to go to the COPR repo and check the spec file:
https://copr-dist-git.fedorainfracloud.org/cgit/msrb/mini-tps/mini-tps.git/t...
which finally told me the URL is https://gitlab.cee.redhat.com/osci/mini-tps . Which...is behind the Red Hat firewall.
So I had to go through like seven steps of investigation to find the actual test code here, and discover that only Red Hat employees can actually access the git repo at all, for a Fedora CI "generic" test.
This seems...not optimal?
On Wed, 2022-01-12 at 09:33 -0800, Adam Williamson wrote:
On Wed, 2022-01-12 at 09:13 -0800, Adam Williamson wrote:
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
Additional: so after writing out this email I thought, hey, I'll go find the source for this test and see if I can fix it.
Whew, boy.
First I went to the CI docs. These do describe the generic tests:
https://docs.fedoraproject.org/en-US/ci/generic_tests/
but provide no information about where the tests actually *come from* (so where you could go to modify them).
Next I checked my Firefox history and remembered there's a Pagure "fedora-ci" team, so I checked the projects under that, but found nothing relevant:
https://pagure.io/group/fedora-ci
so I resorted to web searching. On a search for "fedora rpminspect" (I picked rpminspect as it's a fairly 'unique' name of one of the generic tests) the sixth result was https://github.com/fedora-ci/rpminspect-pipeline , which seems to be the actual implementation of the rpminspect test pipeline. So there's a github fedora-ci as well as the Pagure fedora-ci? Okay. I checked that group's list of projects and found the installability pipeline:
https://github.com/fedora-ci/installability-pipeline
but my search didn't end there, because the test code isn't in the pipeline repo, it pulls it from elsewhere. I had to figure that out. I found it in the 'prepare.sh' script, where it adds a COPR repo and installs a package:
https://github.com/fedora-ci/installability-pipeline/blob/master/prepare.sh#...
so that told me the actual test code was coming from this 'mini-tps' package in this COPR repo. So now I had to go to the COPR repo and check the spec file:
https://copr-dist-git.fedorainfracloud.org/cgit/msrb/mini-tps/mini-tps.git/t...
which finally told me the URL is https://gitlab.cee.redhat.com/osci/mini-tps . Which...is behind the Red Hat firewall.
So I had to go through like seven steps of investigation to find the actual test code here, and discover that only Red Hat employees can actually access the git repo at all, for a Fedora CI "generic" test.
This seems...not optimal?
Welp, never mind, seems that was all a wild goose chase. I think I got the wrong pipeline. I don't know what the installability pipeline is for or when (if ever) it's run, but from the Jenkins log:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipel...
it looks like this test was actually part of the "dist-git" pipeline:
https://github.com/fedora-ci/dist-git-pipeline
but I have no idea how to figure out what that pipeline runs. So I just can't find the code for this test at all. I guess this is the same problem Vit Ondruch ran into recently, except in this case, it doesn't appear that the test itself helpfully logs what it was that got run and how.
On Wed, Jan 12, 2022 at 7:09 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Wed, 2022-01-12 at 09:33 -0800, Adam Williamson wrote:
On Wed, 2022-01-12 at 09:13 -0800, Adam Williamson wrote:
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
Additional: so after writing out this email I thought, hey, I'll go find the source for this test and see if I can fix it.
Whew, boy.
First I went to the CI docs. These do describe the generic tests:
https://docs.fedoraproject.org/en-US/ci/generic_tests/
but provide no information about where the tests actually *come from* (so where you could go to modify them).
Next I checked my Firefox history and remembered there's a Pagure "fedora-ci" team, so I checked the projects under that, but found nothing relevant:
https://pagure.io/group/fedora-ci
so I resorted to web searching. On a search for "fedora rpminspect" (I picked rpminspect as it's a fairly 'unique' name of one of the generic tests) the sixth result was https://github.com/fedora-ci/rpminspect-pipeline , which seems to be the actual implementation of the rpminspect test pipeline. So there's a github fedora-ci as well as the Pagure fedora-ci? Okay. I checked that group's list of projects and found the installability pipeline:
https://github.com/fedora-ci/installability-pipeline
but my search didn't end there, because the test code isn't in the pipeline repo, it pulls it from elsewhere. I had to figure that out. I found it in the 'prepare.sh' script, where it adds a COPR repo and installs a package:
https://github.com/fedora-ci/installability-pipeline/blob/master/prepare.sh#...
so that told me the actual test code was coming from this 'mini-tps' package in this COPR repo. So now I had to go to the COPR repo and check the spec file:
https://copr-dist-git.fedorainfracloud.org/cgit/msrb/mini-tps/mini-tps.git/t...
which finally told me the URL is https://gitlab.cee.redhat.com/osci/mini-tps . Which...is behind the Red Hat firewall.
So I had to go through like seven steps of investigation to find the actual test code here, and discover that only Red Hat employees can actually access the git repo at all, for a Fedora CI "generic" test.
This seems...not optimal?
Welp, never mind, seems that was all a wild goose chase. I think I got the wrong pipeline. I don't know what the installability pipeline is for or when (if ever) it's run, but from the Jenkins log:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipel...
it looks like this test was actually part of the "dist-git" pipeline:
https://github.com/fedora-ci/dist-git-pipeline
but I have no idea how to figure out what that pipeline runs. So I just can't find the code for this test at all. I guess this is the same problem Vit Ondruch ran into recently, except in this case, it doesn't appear that the test itself helpfully logs what it was that got run and how.
The tests are actually in the lorax dist-git repo: https://src.fedoraproject.org/rpms/lorax/blob/f35/f/tests
However, you're probably looking for that package installation part that is currently failing in CI. This is a built-in feature of Testing Farm, so the code is probably somewhere here: https://gitlab.com/testing-farm Unfortunately, I am not familiar with the codebase, so I don't know where to look exactly.
Thanks, Michal
-- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Wed, 2022-01-12 at 21:10 +0100, Michal Srb wrote:
The tests are actually in the lorax dist-git repo: https://src.fedoraproject.org/rpms/lorax/blob/f35/f/tests
However, you're probably looking for that package installation part that is currently failing in CI. This is a built-in feature of Testing Farm, so the code is probably somewhere here: https://gitlab.com/testing-farm Unfortunately, I am not familiar with the codebase, so I don't know where to look exactly.
Yeah, exactly, that's the bit I was looking for. I guess as you say it's technically part of the setup code rather than a 'test', although it winds up acting as a (somewhat broken, as we discussed) test of installability, which is why I got thrown off course when I saw an 'installability' pipeline :D
Yes, we should finally just finally print the repo we tested in our artifacts:
https://src.fedoraproject.org/rpms/lorax/blob/rawhide/f/tests
Sorry this work on improving our main output is getting always pushed back on something :(
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Will do my best add it there finally
/M
On Wed, Jan 12, 2022 at 7:09 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Wed, 2022-01-12 at 09:33 -0800, Adam Williamson wrote:
On Wed, 2022-01-12 at 09:13 -0800, Adam Williamson wrote:
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
Additional: so after writing out this email I thought, hey, I'll go find the source for this test and see if I can fix it.
Whew, boy.
First I went to the CI docs. These do describe the generic tests:
https://docs.fedoraproject.org/en-US/ci/generic_tests/
but provide no information about where the tests actually *come from* (so where you could go to modify them).
Next I checked my Firefox history and remembered there's a Pagure "fedora-ci" team, so I checked the projects under that, but found nothing relevant:
https://pagure.io/group/fedora-ci
so I resorted to web searching. On a search for "fedora rpminspect" (I picked rpminspect as it's a fairly 'unique' name of one of the generic tests) the sixth result was https://github.com/fedora-ci/rpminspect-pipeline , which seems to be the actual implementation of the rpminspect test pipeline. So there's a github fedora-ci as well as the Pagure fedora-ci? Okay. I checked that group's list of projects and found the installability pipeline:
https://github.com/fedora-ci/installability-pipeline
but my search didn't end there, because the test code isn't in the pipeline repo, it pulls it from elsewhere. I had to figure that out. I found it in the 'prepare.sh' script, where it adds a COPR repo and installs a package:
https://github.com/fedora-ci/installability-pipeline/blob/master/prepare.sh#...
so that told me the actual test code was coming from this 'mini-tps' package in this COPR repo. So now I had to go to the COPR repo and check the spec file:
https://copr-dist-git.fedorainfracloud.org/cgit/msrb/mini-tps/mini-tps.git/t...
which finally told me the URL is https://gitlab.cee.redhat.com/osci/mini-tps . Which...is behind the Red Hat firewall.
So I had to go through like seven steps of investigation to find the actual test code here, and discover that only Red Hat employees can actually access the git repo at all, for a Fedora CI "generic" test.
This seems...not optimal?
Welp, never mind, seems that was all a wild goose chase. I think I got the wrong pipeline. I don't know what the installability pipeline is for or when (if ever) it's run, but from the Jenkins log:
https://osci-jenkins-1.ci.fedoraproject.org/job/fedora-ci/job/dist-git-pipel...
it looks like this test was actually part of the "dist-git" pipeline:
https://github.com/fedora-ci/dist-git-pipeline
but I have no idea how to figure out what that pipeline runs. So I just can't find the code for this test at all. I guess this is the same problem Vit Ondruch ran into recently, except in this case, it doesn't appear that the test itself helpfully logs what it was that got run and how. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Hi Adam,
On Wed, Jan 12, 2022 at 6:14 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Wed, 2022-01-12 at 11:47 +0100, Miroslav Vadkerti wrote:
On Tue, Jan 11, 2022 at 10:30 PM Adam Williamson <
adamwill@fedoraproject.org>
wrote:
On Tue, 2022-01-11 at 21:15 +0100, Miroslav Vadkerti wrote:
Hi Adam,
More information is in Testing Farm artifacts, you can find the link
in
Jenkins log:
http://artifacts.dev.testing-farm.io/7d699360-cd4e-4307-b17a-d76424461732/
Seems like some conflict:
Error: Problem: package pipewire-plugin-jack-0.3.43-1.fc35.x86_64 requires jack-audio-connection-kit, but none of the providers can be installed
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.x86_64
- package pipewire-jack-audio-connection-kit-0.3.43-1.fc35.x86_64
conflicts with jack-audio-connection-kit provided by jack-audio-connection-kit-1.9.19-1.fc35.i686
- conflicting requests
Ah, OK, thanks, I see. So that's just this test not really coping with an update containing packages that conflict with default-installed packages, I guess...it's trying to assert that it can install the packages from the update, but in this case it's correct that it can't install pipewire-j-a-c-k since regular j-a-c-k is already installed.
The user has to cope with conflicting packages himself, we are able to exclude some packages from updating using prepare's exclude field, something we use for example in curl:
https://src.fedoraproject.org/rpms/curl/pull-request/11#_2__6
Which has conflicting packages. Maybe that would help here also, maybe not, not sure ...
If there is a way how CI should better handle the update here, we will be happy to fix it.
Well, I guess the interesting wrinkle here is exactly how the test operated. The update in question contained two packages, pipewire and lorax:
https://bodhi.fedoraproject.org/updates/FEDORA-2022-4cf1bb6eed
of those two packages, it's lorax that lists this test as gating in its config file:
https://src.fedoraproject.org/rpms/lorax/blob/rawhide/f/gating.yaml#_7
pipewire does not. And the test reports itself as having run on the lorax package, not on the update. In resultsdb the 'type' is 'koji_build' (i.e. a single Koji build, not a Bodhi update) and the 'item' is 'lorax-35.9-2.fc35'. In CI messages, the 'artifact' dict is all about the 'koji-build' 'lorax-35.9-2.fc35':
https://apps.fedoraproject.org/datagrepper/id?id=2022-eaec10eb-2ecb-4055-8e6...
however, it seems this specific test actually runs as a test of the entire update. It pulls in all the packages from the update and fails if it can't install any of them.
So...we're reporting a failure of a test of "the lorax package" which didn't actually have anything to do with the lorax package. And we're using the gating configuration of the lorax package to decide whether to gate on a test which actually runs against *every package in the update*. From pipewire's perspective, there's no way it can communicate "this test isn't safe for my package, don't run/gate on it" that would work if some *other* package in the update specifies that test as a gating test...
Given the above, I'm not sure we can really say it's ever entirely safe for a package to gate on this test. Because this test is going to function differently depending on what *other* packages are in the same update. Even if a package can affirm that *its own subpackages* should always be installable in any update it's a part of, it can never safely affirm that *all subpackages of any other package in the same update* should always be installable.
I guess this is one of those early design decisions that did not age very well. CI is trying to install all (sub)packages for given builds automatically, before running tests. This means that for example lorax tests don't need to explicitly declare that a specific lorax subpackage is needed, because (in theory) all subpackages should be pre-installed on the test system automatically. And this works surprisingly well for the vast majority of packages as they don't have any conflicts on subpackage level. Maybe that's why this CI behavior seemed like a good idea initially. I am actually not sure when this behavior was introduced, but it's been the default in CI for a very long time now.
I personally wouldn't be opposed to a nuclear solution when we would simply change this behavior (with some delay after the announcement) so that all tests would need to properly declare all their dependencies.
Thanks, Michal
I guess the smallest 'fix' for this would be to make all packages from the update *available* to dnf during this test (in case the package under test has interdependencies with the other packages in the update, which is likely why they're bundled together), but only have it assert that *subpackages of the specific package under test* are installable. As long as the test is claimed to be a test of the *package* and not the *update*, it shouldn't fail if a subpackage of *some other package in the update* is not installable.
WDYT?
Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 12. 01. 22 21:00, Michal Srb wrote:
I guess this is one of those early design decisions that did not age very well. CI is trying to install all (sub)packages for given builds automatically, before running tests. This means that for example lorax tests don't need to explicitly declare that a specific lorax subpackage is needed, because (in theory) all subpackages should be pre-installed on the test system automatically. And this works surprisingly well for the vast majority of packages as they don't have any conflicts on subpackage level. Maybe that's why this CI behavior seemed like a good idea initially. I am actually not sure when this behavior was introduced, but it's been the default in CI for a very long time now.
Relevant issue from 2019:
"RFE: Don't install all sub-packages into the testing environment"
https://pagure.io/fedora-ci/general/issue/29
However in this case it's even worse. It installs all *other* packages from that update, which does not seem like a good idea even for vast majority of the packages.
I personally wouldn't be opposed to a nuclear solution when we would simply change this behavior (with some delay after the announcement) so that all tests would need to properly declare all their dependencies.
+1
Hi,
On Wed, Jan 12, 2022 at 9:12 PM Miro Hrončok mhroncok@redhat.com wrote:
On 12. 01. 22 21:00, Michal Srb wrote:
I guess this is one of those early design decisions that did not age
very well.
CI is trying to install all (sub)packages for given builds
automatically,
before running tests. This means that for example lorax tests don't need
to
explicitly declare that a specific lorax subpackage is needed, because
(in
theory) all subpackages should be pre-installed on the test system automatically. And this works surprisingly well for the vast majority of packages as they don't have any conflicts on subpackage level. Maybe
that's why
this CI behavior seemed like a good idea initially. I am actually not
sure when
this behavior was introduced, but it's been the default in CI for a very
long
time now.
Relevant issue from 2019:
"RFE: Don't install all sub-packages into the testing environment"
Commented, with package excluding most of the packages should be unblocked.
However in this case it's even worse. It installs all *other* packages from that update, which does not seem like a good idea even for vast majority of the packages.
I personally wouldn't be opposed to a nuclear solution when we would
simply
change this behavior (with some delay after the announcement) so that
all tests
would need to properly declare all their dependencies.
+1
Well we are fine to do that change, but I will find another couple of folks who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build, so feels like we are missing the "install only these (sub)packages" scenario ...
Can somebody give me a package where this would be handy?
Thanks, /M
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, 2022-01-13 at 18:04 +0100, Miroslav Vadkerti wrote:
Well we are fine to do that change, but I will find another couple of folks who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build, so feels like we are missing the "install only these (sub)packages" scenario ...
Can somebody give me a package where this would be handy?
Well, the question I'd have then is, who should have specified that pipewire should be "excluded", and where? It doesn't really make sense to expect that lorax should specify this in its test config, does it? The fact that we had to put lorax and pipewire together in an update was a kind of unusual and unforeseen circumstance. Because of its nature, lorax could potentially be combined with all sorts of other packages in an update. It's not realistic for lorax to 'know' domain- specific test config information for any other package it could possibly wind up bundled with in an update.
So, it'd be more sensible for pipewire to do it, but...we're still treating gating as optional at the moment, and pipewire isn't opting into it. So again it seems a bit odd for us to say 'hey, pipewire, you know that CI gating thing you're not opting into? You need to carry some configuration for it just in case anything else that does opt into CI gating is ever bundled with pipewire in an update'.
For me, at a minimum, since the system is in general claiming to be operating on the level of *the package* not *the update*, it should only expressly install the subpackages of the package it claims to be operating on. If they depend on subpackages of other packages in the update, then of course those dependencies should be expressed and those other subpackages will get pulled in automatically. But CI shouldn't just be expressly installing everything in the update when it's only claiming to be operating on one package from it.
Not expressly installing anything at all feels like a bigger step; it might be the right thing to do, but it would definitely be a 'flag day' kinda thing. But it seems to me that my suggestion should have a much smaller impact.
On Thu, Jan 13, 2022 at 6:37 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Thu, 2022-01-13 at 18:04 +0100, Miroslav Vadkerti wrote:
Well we are fine to do that change, but I will find another couple of
folks
who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build, so feels like we are missing the "install only these (sub)packages" scenario ...
Can somebody give me a package where this would be handy?
Well, the question I'd have then is, who should have specified that pipewire should be "excluded", and where? It doesn't really make sense to expect that lorax should specify this in its test config, does it? The fact that we had to put lorax and pipewire together in an update was a kind of unusual and unforeseen circumstance. Because of its nature, lorax could potentially be combined with all sorts of other packages in an update. It's not realistic for lorax to 'know' domain- specific test config information for any other package it could possibly wind up bundled with in an update.
So, it'd be more sensible for pipewire to do it, but...we're still treating gating as optional at the moment, and pipewire isn't opting into it. So again it seems a bit odd for us to say 'hey, pipewire, you know that CI gating thing you're not opting into? You need to carry some configuration for it just in case anything else that does opt into CI gating is ever bundled with pipewire in an update'.
For me, at a minimum, since the system is in general claiming to be operating on the level of *the package* not *the update*, it should only expressly install the subpackages of the package it claims to be operating on. If they depend on subpackages of other packages in the update, then of course those dependencies should be expressed and those other subpackages will get pulled in automatically. But CI shouldn't just be expressly installing everything in the update when it's only claiming to be operating on one package from it.
Yep, I agree, installing both seems like an un-needed evil, so we will prepare a repo with both in the system, but install only lorax for the lorax run. For the pipewire we will install only pipewire, and have lorax around in the artifact repo, makes sense to mee.
Thanks for the suggestion, let's make things better!
Best regards, /M
Not expressly installing anything at all feels like a bigger step; it might be the right thing to do, but it would definitely be a 'flag day' kinda thing. But it seems to me that my suggestion should have a much smaller impact. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Thu, Jan 13, 2022 at 6:58 PM Miroslav Vadkerti mvadkert@redhat.com wrote:
On Thu, Jan 13, 2022 at 6:37 PM Adam Williamson < adamwill@fedoraproject.org> wrote:
On Thu, 2022-01-13 at 18:04 +0100, Miroslav Vadkerti wrote:
Well we are fine to do that change, but I will find another couple of
folks
who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build,
so
feels like we are missing the "install only these (sub)packages"
scenario
...
Can somebody give me a package where this would be handy?
Well, the question I'd have then is, who should have specified that pipewire should be "excluded", and where? It doesn't really make sense to expect that lorax should specify this in its test config, does it? The fact that we had to put lorax and pipewire together in an update was a kind of unusual and unforeseen circumstance. Because of its nature, lorax could potentially be combined with all sorts of other packages in an update. It's not realistic for lorax to 'know' domain- specific test config information for any other package it could possibly wind up bundled with in an update.
So, it'd be more sensible for pipewire to do it, but...we're still treating gating as optional at the moment, and pipewire isn't opting into it. So again it seems a bit odd for us to say 'hey, pipewire, you know that CI gating thing you're not opting into? You need to carry some configuration for it just in case anything else that does opt into CI gating is ever bundled with pipewire in an update'.
For me, at a minimum, since the system is in general claiming to be operating on the level of *the package* not *the update*, it should only expressly install the subpackages of the package it claims to be operating on. If they depend on subpackages of other packages in the update, then of course those dependencies should be expressed and those other subpackages will get pulled in automatically. But CI shouldn't just be expressly installing everything in the update when it's only claiming to be operating on one package from it.
Yep, I agree, installing both seems like an un-needed evil, so we will prepare a repo with both in the system, but install only lorax for the lorax run. For the pipewire we will install only pipewire, and have lorax around in the artifact repo, makes sense to mee.
Thanks for the suggestion, let's make things better!
This should give us API support:
https://gitlab.com/testing-farm/api/-/merge_requests/23
@msrb idea is that you will use this field to indicate which artifacts you want to skip from installation. I will try to send a patch for this ...
/M
Best regards, /M
Not expressly installing anything at all feels like a bigger step; it might be the right thing to do, but it would definitely be a 'flag day' kinda thing. But it seems to me that my suggestion should have a much smaller impact. -- Adam Williamson Fedora QA IRC: adamw | Twitter: adamw_ha https://www.happyassassin.net
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o
št 13. 1. 2022 o 18:07 Miroslav Vadkerti mvadkert@redhat.com napísal(a):
Hi,
On Wed, Jan 12, 2022 at 9:12 PM Miro Hrončok mhroncok@redhat.com wrote:
On 12. 01. 22 21:00, Michal Srb wrote:
I guess this is one of those early design decisions that did not age
very well.
CI is trying to install all (sub)packages for given builds
automatically,
before running tests. This means that for example lorax tests don't
need to
explicitly declare that a specific lorax subpackage is needed, because
(in
theory) all subpackages should be pre-installed on the test system automatically. And this works surprisingly well for the vast majority
of
packages as they don't have any conflicts on subpackage level. Maybe
that's why
this CI behavior seemed like a good idea initially. I am actually not
sure when
this behavior was introduced, but it's been the default in CI for a
very long
time now.
Relevant issue from 2019:
"RFE: Don't install all sub-packages into the testing environment"
Commented, with package excluding most of the packages should be unblocked.
However in this case it's even worse. It installs all *other* packages from that update, which does not seem like a good idea even for vast majority of the packages.
I personally wouldn't be opposed to a nuclear solution when we would
simply
change this behavior (with some delay after the announcement) so that
all tests
would need to properly declare all their dependencies.
+1
Well we are fine to do that change, but I will find another couple of folks who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build, so feels like we are missing the "install only these (sub)packages" scenario ...
I can think of 2 (more) use cases where the default install-everything behavior can be problematic.
* opening a PR against tests/ namespace -- this simply runs the tests on the existing build, no Koji task id is explicitly passed to Testing Farm. So if the tests don't pull in the required packages, CI will fail. It could be argued that this is a bug in CI, but this is how it works now.
* reverse dependency testing -- I have a new build and I want to make sure that packages that depend on my component will still work when I push the build to the repository. So I can tell Testing Farm to run the tests of the dependent components, but I don't necessarily know the NVR of those components.
With all that said, I don't really see a good reason why tests shouldn't properly declare all their dependencies.
Thanks, Michal
Can somebody give me a package where this would be handy?
Thanks, /M
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Tue, Jan 18, 2022 at 11:04 AM Michal Srb msrb@redhat.com wrote:
št 13. 1. 2022 o 18:07 Miroslav Vadkerti mvadkert@redhat.com napísal(a):
Hi,
On Wed, Jan 12, 2022 at 9:12 PM Miro Hrončok mhroncok@redhat.com wrote:
On 12. 01. 22 21:00, Michal Srb wrote:
I guess this is one of those early design decisions that did not age
very well.
CI is trying to install all (sub)packages for given builds
automatically,
before running tests. This means that for example lorax tests don't
need to
explicitly declare that a specific lorax subpackage is needed, because
(in
theory) all subpackages should be pre-installed on the test system automatically. And this works surprisingly well for the vast majority
of
packages as they don't have any conflicts on subpackage level. Maybe
that's why
this CI behavior seemed like a good idea initially. I am actually not
sure when
this behavior was introduced, but it's been the default in CI for a
very long
time now.
Relevant issue from 2019:
"RFE: Don't install all sub-packages into the testing environment"
Commented, with package excluding most of the packages should be unblocked.
However in this case it's even worse. It installs all *other* packages from that update, which does not seem like a good idea even for vast majority of the packages.
I personally wouldn't be opposed to a nuclear solution when we would
simply
change this behavior (with some delay after the announcement) so that
all tests
would need to properly declare all their dependencies.
+1
Well we are fine to do that change, but I will find another couple of folks who think the default install all is good :( So maybe we just need to be able to fine-tune it. We now can exclude packages from the koji build, so feels like we are missing the "install only these (sub)packages" scenario ...
I can think of 2 (more) use cases where the default install-everything behavior can be problematic.
- opening a PR against tests/ namespace -- this simply runs the tests on
the existing build, no Koji task id is explicitly passed to Testing Farm. So if the tests don't pull in the required packages, CI will fail. It could be argued that this is a bug in CI, but this is how it works now.
Right, tests should require what they need for testing, there are no objections about that. If they don't, they fail but it is not a bug in CI.
- reverse dependency testing -- I have a new build and I want to make sure
that packages that depend on my component will still work when I push the build to the repository. So I can tell Testing Farm to run the tests of the dependent components, but I don't necessarily know the NVR of those components.
With all that said, I don't really see a good reason why tests shouldn't properly declare all their dependencies.
I am not advocating for going against that idea, I am just saying if they don't, we now have to install everything now. The reason is, that artifact installation is currently Testing Farm's responsibility. Once we drop this requirement and make the artifact installation part of the responsibility of tmt, we can drop that. The API now clearly states:
artifacts - Array of objects (environment.artifacts) - Additional artifacts to install in the test environment.
I guess we should file an issue somewhere ...
Best regards, /M
Thanks, Michal
Can somebody give me a package where this would be handy?
Thanks, /M
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / Linux QE IRC mvadkert #tft #tmt #osci :: Mobile +420 773 944 252 Remote Czech Republic :: Red Hat Czech s.r.o
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
CI mailing list -- ci@lists.fedoraproject.org To unsubscribe send an email to ci-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/ci@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
-
Adam Williamson -
Michal Srb -
Miro Hrončok -
Miroslav Vadkerti