Is packit suitable for tests (kdump tests) that need to start a VM via QEMU and then check the console output?
Hi,
I'm adding some kdump tests to dracut to prevent dracut's change from breaking kexec-tools [1]. One suggestion is to make use of packit. The kdump tests are different from other tests because we need to reboot into the second kernel (a dump-capture kernel) to dump the system kernel's memory locally fs or to a remote fs (e.g. an nfs fs). Take the nfs-kdump test [2] as an example, here are what this test does, 1. Build the rpms for dracut and kexec-tools 2. Download the Fedora 33 Cloud Base Image 3. Start NFS server in a VM 4. For the kdump tests done as an NFS client, - Install the rpms onto another copy of Fedora 33 Cloud Base Image - Start another VM and trigger sysrq to boot into the dump-capture kernel to do vmcore dump - Check the console output to see if vmcore dump is successful
I'm not sure packit is suitable for these kdump tests. For example, I need to get the path of the rpm built automatically by packit so the package can be installed onto the VM image but nowhere can I find anything related to that in packit's docs [3] or [4].
[1] https://github.com/dracutdevs/dracut/pull/1163 [2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests/scripts/test... [3] https://packit.dev/docs/testing-farm/ [4] https://docs.fedoraproject.org/en-US/ci/
On Tue, Apr 06, 2021 at 07:36:49AM +0800, Coiby Xu wrote:
Hi,
I'm adding some kdump tests to dracut to prevent dracut's change from breaking kexec-tools [1]. One suggestion is to make use of packit. The kdump tests are different from other tests because we need to reboot into the second kernel (a dump-capture kernel) to dump the system kernel's memory locally fs or to a remote fs (e.g. an nfs fs). Take the nfs-kdump test [2] as an example, here are what this test does,
- Build the rpms for dracut and kexec-tools
- Download the Fedora 33 Cloud Base Image
- Start NFS server in a VM
- For the kdump tests done as an NFS client,
- Install the rpms onto another copy of Fedora 33 Cloud Base Image
- Start another VM and trigger sysrq to boot into the dump-capture kernel to do vmcore dump
- Check the console output to see if vmcore dump is successful
I'm not sure packit is suitable for these kdump tests. For example, I need to get the path of the rpm built automatically by packit so the package can be installed onto the VM image but nowhere can I find anything related to that in packit's docs [3] or [4].
When I checked https://prod.packit.dev/copr-build/96803, I found the answer, i.e. "dnf copr enable packit/dracutdevs-dracut-1163":)
Then here's another question, can I test against the Atomic Host test subject directly? I imagine packit will also start a VM to run the tests. If we run these kdump tests, nested VMs will be started. Can I eliminate the nested VMS with the help from packit?
[1] https://github.com/dracutdevs/dracut/pull/1163 [2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests/scripts/test... [3] https://packit.dev/docs/testing-farm/ [4] https://docs.fedoraproject.org/en-US/ci/
-- Best regards, Coiby
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
More comments inline.
On Tue, Apr 6, 2021 at 2:39 AM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 07:36:49AM +0800, Coiby Xu wrote:
Hi,
I'm adding some kdump tests to dracut to prevent dracut's change from breaking kexec-tools [1]. One suggestion is to make use of packit. The kdump tests are different from other tests because we need to reboot into the second kernel (a dump-capture kernel) to dump the system kernel's memory locally fs or to a remote fs (e.g. an nfs fs). Take the nfs-kdump test [2] as an example, here are what this test does,
- Build the rpms for dracut and kexec-tools
- Download the Fedora 33 Cloud Base Image
- Start NFS server in a VM
- For the kdump tests done as an NFS client,
- Install the rpms onto another copy of Fedora 33 Cloud Base Image
- Start another VM and trigger sysrq to boot into the dump-capture kernel to do vmcore dump
- Check the console output to see if vmcore dump is successful
Petr and Miro, would this workflow work with tmt and TF?
I'm not sure packit is suitable for these kdump tests. For example, I need to get the path of the rpm built automatically by packit so the package can be installed onto the VM image but nowhere can I find anything related to that in packit's docs [3] or [4].
We can provide that information via env vars (that would be an RFE for packit) or you can discover this by querying the RPM database on the testing system.
When I checked https://prod.packit.dev/copr-build/96803, I found the answer, i.e. "dnf copr enable packit/dracutdevs-dracut-1163":)
This is correct, by default packit creates a new COPR project for every PR using such scheme - you can also build in your own COPR project if that makes things easier for you.
Then here's another question, can I test against the Atomic Host test subject directly? I imagine packit will also start a VM to run the tests. If we run these kdump tests, nested VMs will be started. Can I eliminate the nested VMS with the help from packit?
AFAIK we have never tried that. Isn't Atomic Host obsolete at this point and being replaced by Fedora CoreOS? What's exactly your use case to test against Atomic Host?
https://www.projectatomic.io/blog/2019/11/fedora-atomic-host-nearing-eol/
Regards, Tomas
Best regards, Coiby
User-cont-team mailing list User-cont-team@redhat.com https://listman.redhat.com/mailman/listinfo/user-cont-team
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
More comments inline.
On Tue, Apr 6, 2021 at 2:39 AM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 07:36:49AM +0800, Coiby Xu wrote:
Hi,
I'm adding some kdump tests to dracut to prevent dracut's change from breaking kexec-tools [1]. One suggestion is to make use of packit. The kdump tests are different from other tests because we need to reboot into the second kernel (a dump-capture kernel) to dump the system kernel's memory locally fs or to a remote fs (e.g. an nfs fs). Take the nfs-kdump test [2] as an example, here are what this test does,
- Build the rpms for dracut and kexec-tools
- Download the Fedora 33 Cloud Base Image
- Start NFS server in a VM
- For the kdump tests done as an NFS client,
- Install the rpms onto another copy of Fedora 33 Cloud Base Image
- Start another VM and trigger sysrq to boot into the dump-capture kernel to do vmcore dump
- Check the console output to see if vmcore dump is successful
Petr and Miro, would this workflow work with tmt and TF?
I'm not sure packit is suitable for these kdump tests. For example, I need to get the path of the rpm built automatically by packit so the package can be installed onto the VM image but nowhere can I find anything related to that in packit's docs [3] or [4].
We can provide that information via env vars (that would be an RFE for packit) or you can discover this by querying the RPM database on the testing system.
Are those env vars available now? If so, can you point me to the docs?
When I checked https://prod.packit.dev/copr-build/96803, I found the answer, i.e. "dnf copr enable packit/dracutdevs-dracut-1163":)
This is correct, by default packit creates a new COPR project for every PR using such scheme - you can also build in your own COPR project if that makes things easier for you.
For now, env vars seems to be a better choice:)
Then here's another question, can I test against the Atomic Host test subject directly? I imagine packit will also start a VM to run the tests. If we run these kdump tests, nested VMs will be started. Can I eliminate the nested VMS with the help from packit?
AFAIK we have never tried that. Isn't Atomic Host obsolete at this point and being replaced by Fedora CoreOS? What's exactly your use case to test against Atomic Host?
https://www.projectatomic.io/blog/2019/11/fedora-atomic-host-nearing-eol/
I used Atomic Host as an example because it's listed on [1] as one of the three test subjects and kdump tests are done using a Fedora Cloud Base Image which is similar to Atomic Host. So I guess packit and the kdump tests framework share some code on manipulating a VM image and I wonder if the kdump tests can be done without nested VMs with the help of packit.
[1] https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
Regards, Tomas
Best regards, Coiby
User-cont-team mailing list User-cont-team@redhat.com https://listman.redhat.com/mailman/listinfo/user-cont-team
On Tue, Apr 6, 2021 at 11:28 AM Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
We can provide that information via env vars (that would be an RFE for packit) or you can discover this by querying the RPM database on the testing system.
Are those env vars available now? If so, can you point me to the docs?
As I said, that would be an RFE - we'd need to implement it first :)
Just to make it clear: the ask is to provide name-version-release or path of the installed RPMs in the testing environment, do I get this right? So that you can utilize the precise NVR of the built RPMs in your tests and install them in VMs of your choice.
I'm gonna create an issue to capture this request once you confirm my explanation above.
Tomas
On Tue, Apr 06, 2021 at 03:29:51PM +0200, Tomas Tomecek wrote:
On Tue, Apr 6, 2021 at 11:28 AM Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
We can provide that information via env vars (that would be an RFE for packit) or you can discover this by querying the RPM database on the testing system.
Are those env vars available now? If so, can you point me to the docs?
As I said, that would be an RFE - we'd need to implement it first :)
Now I understand what RFE means :)
Just to make it clear: the ask is to provide name-version-release or path of the installed RPMs in the testing environment, do I get this right? So that you can utilize the precise NVR of the built RPMs in your tests and install them in VMs of your choice.
I'm not sure what name-version-release means exactly. But a path of the installed RPMs sounds good.
I'm gonna create an issue to capture this request once you confirm my explanation above.
Tomas
On Wed, Apr 7, 2021 at 1:30 PM Coiby Xu coxu@redhat.com wrote:
Are those env vars available now? If so, can you point me to the docs?
As I said, that would be an RFE - we'd need to implement it first :)
Now I understand what RFE means :)
Sorry for using shortcuts without explaining them. RFE stands for "request for enhancement" and I am using that term often for feature requests.
Just to make it clear: the ask is to provide name-version-release or path of the installed RPMs in the testing environment, do I get this right? So that you can utilize the precise NVR of the built RPMs in your tests and install them in VMs of your choice.
I'm not sure what name-version-release means exactly. But a path of the installed RPMs sounds good.
Name-version-release is the term we use in the context of a buildsystem to uniquely identify precise builds.
If I pick a kernel build such as this one: https://koji.fedoraproject.org/koji/buildinfo?buildID=1732452
kernel-5.12.0-0.rc6.184.fc35
kernel - name 5.12.0 - version 0.rc6.184.fc35 - release
you often find this term shortened as NVR
I'm gonna create an issue to capture this request once you confirm my explanation above.
Created an issue to track this: https://github.com/packit/packit-service/issues/1045
Tomas
On Wed, Apr 07, 2021 at 01:43:48PM +0200, Tomas Tomecek wrote:
On Wed, Apr 7, 2021 at 1:30 PM Coiby Xu coxu@redhat.com wrote:
Are those env vars available now? If so, can you point me to the docs?
As I said, that would be an RFE - we'd need to implement it first :)
Now I understand what RFE means :)
Sorry for using shortcuts without explaining them. RFE stands for "request for enhancement" and I am using that term often for feature requests.
Just to make it clear: the ask is to provide name-version-release or path of the installed RPMs in the testing environment, do I get this right? So that you can utilize the precise NVR of the built RPMs in your tests and install them in VMs of your choice.
I'm not sure what name-version-release means exactly. But a path of the installed RPMs sounds good.
Name-version-release is the term we use in the context of a buildsystem to uniquely identify precise builds.
If I pick a kernel build such as this one: https://koji.fedoraproject.org/koji/buildinfo?buildID=1732452
kernel-5.12.0-0.rc6.184.fc35
kernel - name 5.12.0 - version 0.rc6.184.fc35 - release
you often find this term shortened as NVR
Thanks for clarifying what RFE and NVR means for me!
I'm gonna create an issue to capture this request once you confirm my explanation above.
Created an issue to track this: https://github.com/packit/packit-service/issues/1045
Thanks! I've subscribed to this issue.
Tomas
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
- https://github.com/psss/tmt/issues/281
Reboot is possible using an ansible playbook during the prepare step:
- https://tmt.readthedocs.io/en/latest/spec/plans.html#ansible
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
psss...
More comments inline.
On Tue, Apr 6, 2021 at 2:39 AM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 07:36:49AM +0800, Coiby Xu wrote:
Hi,
I'm adding some kdump tests to dracut to prevent dracut's change from breaking kexec-tools [1]. One suggestion is to make use of packit. The kdump tests are different from other tests because we need to reboot into the second kernel (a dump-capture kernel) to dump the system kernel's memory locally fs or to a remote fs (e.g. an nfs fs). Take the nfs-kdump test [2] as an example, here are what this test does,
- Build the rpms for dracut and kexec-tools
- Download the Fedora 33 Cloud Base Image
- Start NFS server in a VM
- For the kdump tests done as an NFS client,
- Install the rpms onto another copy of Fedora 33 Cloud Base Image
- Start another VM and trigger sysrq to boot into the dump-capture kernel to do vmcore dump
- Check the console output to see if vmcore dump is successful
Petr and Miro, would this workflow work with tmt and TF?
I'm not sure packit is suitable for these kdump tests. For example, I need to get the path of the rpm built automatically by packit so the package can be installed onto the VM image but nowhere can I find anything related to that in packit's docs [3] or [4].
We can provide that information via env vars (that would be an RFE for packit) or you can discover this by querying the RPM database on the testing system.
Are those env vars available now? If so, can you point me to the docs?
When I checked https://prod.packit.dev/copr-build/96803, I found the answer, i.e. "dnf copr enable packit/dracutdevs-dracut-1163":)
This is correct, by default packit creates a new COPR project for every PR using such scheme - you can also build in your own COPR project if that makes things easier for you.
For now, env vars seems to be a better choice:)
Then here's another question, can I test against the Atomic Host test subject directly? I imagine packit will also start a VM to run the
tests.
If we run these kdump tests, nested VMs will be started. Can I eliminate the nested VMS with the help from packit?
AFAIK we have never tried that. Isn't Atomic Host obsolete at this point and being replaced by Fedora CoreOS? What's exactly your use case to test against Atomic Host?
https://www.projectatomic.io/blog/2019/11/fedora-atomic-host-nearing-eol/
I used Atomic Host as an example because it's listed on [1] as one of the three test subjects and kdump tests are done using a Fedora Cloud Base Image which is similar to Atomic Host. So I guess packit and the kdump tests framework share some code on manipulating a VM image and I wonder if the kdump tests can be done without nested VMs with the help of packit.
[1] https://docs.fedoraproject.org/en-US/ci/standard-test-roles/
Regards, Tomas
Best regards, Coiby
User-cont-team mailing list User-cont-team@redhat.com https://listman.redhat.com/mailman/listinfo/user-cont-team
-- Best regards, Coiby
Hi Petr,
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
[...]
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
Sorry for not having been clear before. For now, the kdump tests will automatically reboot the VM after finishing vmcore dumping in the dump-capture kernel. So reboot during the test execution is not needed currently.
What's required is if there is a way to get the console output because the tests rely on the console output to know if the tests are successful.
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Yes, this is another must-have feature. Without this feature, only the test of dumping vmcore locally can be done. Because the rest of the tests need a second VM to act as a SSH or NFS server.
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm. Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you share some materials with me since docs.fedoraproject.org/en-US/ci seems to be outdated?
On Tue, 13 Apr 2021 at 12:57, Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm. Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you share some materials with me since docs.fedoraproject.org/en-US/ci seems to be outdated?
Hi Coiby,
the tmt part of the Fedora CI documentation should be up-to-date: https://docs.fedoraproject.org/en-US/ci/tmt/
Docs related to the provision step are located here: https://tmt.readthedocs.io/en/latest/spec/plans.html#provision
Running tests inside a container can be done from the command line using: tmt run --all provision --how container
See also the provision options section for some more examples: https://tmt.readthedocs.io/en/latest/examples.html#provision-options
As for now, tmt itself currently supports only a single guest, not sure how multiple vm instances would work under the container and with the Testing Farm. Miro will know all the details.
We have regular tmt hacking sessions on Tuesday. I'll send you an invite.
psss...
On Wed, Apr 14, 2021 at 01:29:20PM +0200, Petr Šplíchal wrote:
On Tue, 13 Apr 2021 at 12:57, Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm. Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you share some materials with me since docs.fedoraproject.org/en-US/ci seems to be outdated?
Hi Coiby,
the tmt part of the Fedora CI documentation should be up-to-date: https://docs.fedoraproject.org/en-US/ci/tmt/
Docs related to the provision step are located here: https://tmt.readthedocs.io/en/latest/spec/plans.html#provision
Running tests inside a container can be done from the command line using: tmt run --all provision --how container
See also the provision options section for some more examples: https://tmt.readthedocs.io/en/latest/examples.html#provision-options
Thanks for pointing me to these links.
As for now, tmt itself currently supports only a single guest, not sure how multiple vm instances would work under the container and with the Testing Farm. Miro will know all the details.
Github's CI (Actions) runs the tests under the container. What [1] does is to make tests run in Github Actions. I've also successfully run the kdump tests under the container locally. So I think it's also doable in tmt.
[1] https://github.com/dracutdevs/dracut/pull/1163
We have regular tmt hacking sessions on Tuesday. I'll send you an invite.
Thanks!
psss...
Hi Coiby,
On Tue, Apr 13, 2021 at 12:57 PM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm.
I expect in this reply that "manage multiple VM instances by tests" means, that your tests will take care of provisioning of the VMs via available /dev/kvm (e.g using libvirt, qemu-kvm, or similar ...) on the test host ...
If that is not what you are expecting, I guess I will need to reply again ...
....
Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do not support nested virtualization and we do not yet have bare metal support with AWS :/.
So ... the only way to make this work **now** is to use internal infrastructure, but that is outside of Packit, and you would need to call Testing Farm service yourself (from Github actions for example), scripting the communication. That way, you could, from upstream pull request, provision internal infrastructure and return back results. The only drawback of this solution is that the artifacts storage are not accessible from the public, you will just get an xunit XML with Pass/Fail results, but the links will lead to internal Red Hat infrastructure.
Unfortunately, as we do not yet have any users doing exactly this, this would be a prototype for your team. But should be possible ...
If the internal artifact storage is a no-go for you, we do not have a possible solution right now. But we will be glad to talk about possibilities, like prototyping the use of AWS bare metal instances ....
If the problem is that this goes around Packit, we can have a talk if this feature (run tests against internal infrastucture) is something we could also support via Packit. But this also needs some additional work ...
Unfortunately we are a bit behind with docs, so the best sources we now have are:
https://packit.dev/docs/testing-farm/ https://docs.fedoraproject.org/en-US/ci/tmt/
But what I wrote here is not included there ...
Best regards, /M
Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you share some materials with me since docs.fedoraproject.org/en-US/ci seems to be outdated?
Hi Miro,
Thanks for the reply!
On Wed, Apr 14, 2021 at 02:16:02PM +0200, Miroslav Vadkerti wrote:
Hi Coiby,
On Tue, Apr 13, 2021 at 12:57 PM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests. TF itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is actually running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM reboots right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests, an important question is whether TF supports testing VM reboots. So I wait for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use case you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm.
I expect in this reply that "manage multiple VM instances by tests" means, that your tests will take care of provisioning of the VMs via available /dev/kvm (e.g using libvirt, qemu-kvm, or similar ...) on the test host ...
If that is not what you are expecting, I guess I will need to reply again ...
Yes, what the kdump tests [2] do is to use qemu-kvm to start multiple VM instances. This part of work has already be implemented by my collogue. But this is not I'm expecting. I'm afraid you need you reply again.
Since Packit supports running tests inside a VM, I expect packet to provision multiple VM instances for a single test and make the console output of the VMs accessible to test. If this works, one great benefit of using Packit is I can run the sames tests for both upstream and downstream kexec-tools and dracut. Because commits backported from upstream dracut could also break kexec-tools.
Btw, the kdump tests have already implemented a framework to provision multiple VM instances and watch the console output. Maybe I can help implement these features for Packit?
[2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests
....
Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do not support nested virtualization and we do not yet have bare metal support with AWS :/.
I see. So a AWS EC2 instance doesn't support starting a VM via qemu, right?
So ... the only way to make this work **now** is to use internal infrastructure, but that is outside of Packit, and you would need to call Testing Farm service yourself (from Github actions for example), scripting the communication. That way, you could, from upstream pull request, provision internal infrastructure and return back results. The only drawback of this solution is that the artifacts storage are not accessible from the public, you will just get an xunit XML with Pass/Fail results, but the links will lead to internal Red Hat infrastructure.
Unfortunately, as we do not yet have any users doing exactly this, this would be a prototype for your team. But should be possible ...
If the internal artifact storage is a no-go for you, we do not have a possible solution right now. But we will be glad to talk about possibilities, like prototyping the use of AWS bare metal instances ....
If the problem is that this goes around Packit, we can have a talk if this feature (run tests against internal infrastucture) is something we could also support via Packit.
This feature is nice to have but I could prefer what has been described before.
But this also needs some additional work ...
Unfortunately we are a bit behind with docs, so the best sources we now have are:
https://packit.dev/docs/testing-farm/ https://docs.fedoraproject.org/en-US/ci/tmt/
Thanks for recommending these docs!
But what I wrote here is not included there ...
Best regards, /M
Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you share some materials with me since docs.fedoraproject.org/en-US/ci seems to be outdated?
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95 TPB-C 2C215 :: Mobile +420 773 944 252 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
Hi Coiby,
On Wed, Apr 14, 2021 at 3:57 PM Coiby Xu coxu@redhat.com wrote:
Hi Miro,
Thanks for the reply!
On Wed, Apr 14, 2021 at 02:16:02PM +0200, Miroslav Vadkerti wrote:
Hi Coiby,
On Tue, Apr 13, 2021 at 12:57 PM Coiby Xu coxu@redhat.com wrote:
On Tue, Apr 06, 2021 at 05:05:03PM +0200, Petr Šplíchal wrote:
On Tue, 6 Apr 2021 at 11:28, Coiby Xu coxu@redhat.com wrote:
Hi Tomas,
Thank you for the reply!
On Tue, Apr 06, 2021 at 10:19:51AM +0200, Tomas Tomecek wrote:
Hi Coiby, thanks for reaching out!
Packit service is using the Testing Farm (TF) project to run tests.
TF
itself uses tmt and fmf projects [tmt]. So if you want to run your tests in the packit workflow, you'd need to make sure they run correctly with tmt tool itself. I am CCing Petr Splichal and Miro Vadkerti, the lead developers of tmt and TF projects. Petr is
actually
running periodic onboarding sessions where he's helping engineers to onboard their projects. I'm not sure if TF supports testing VM
reboots
right now, I will defer answering that question to the guys.
[tmt] https://docs.fedoraproject.org/en-US/ci/tmt/#_summary https://tmt.readthedocs.io/en/stable/
Last week I played with tmt to run the shell tests. For kdump tests,
an
important question is whether TF supports testing VM reboots. So I
wait
for Petr Splichal and Miro Vadkerti's answer.
Hi!
As for now, reboot during the test execution itself is not supported in tmt. Here's the issue where we discuss the feature:
Reboot is possible using an ansible playbook during the prepare step:
However, I'm not sure this approach would be sufficient for the use
case
you've described (e.g. several vm instances). I guess Miro will provide some more details from the TFT point of view.
Hi Petr,
Before Miro provides more details, I'd like to run the tests inside a docker and manage multiple VM instances by the tests themselves to evaluate the benefits of Testing Farm.
I expect in this reply that "manage multiple VM instances by tests" means, that your tests will take care of provisioning of the VMs via available /dev/kvm (e.g using libvirt, qemu-kvm, or similar ...) on the test host
...
If that is not what you are expecting, I guess I will need to reply again ...
Yes, what the kdump tests [2] do is to use qemu-kvm to start multiple VM instances. This part of work has already be implemented by my collogue. But this is not I'm expecting. I'm afraid you need you reply again.
Ok :)
Since Packit supports running tests inside a VM, I expect packet to provision multiple VM instances for a single test and make the console output of the VMs accessible to test.
Unfortunately multihost support is not yet implemented. All testing we do is single host.
If this works, one great benefit of using Packit is I can run the sames tests for both upstream and downstream kexec-tools and dracut. Because commits backported from upstream dracut could also break kexec-tools.
Yeah, we would like definitely to see multihost supported ...
Btw, the kdump tests have already implemented a framework to provision multiple VM instances and watch the console output. Maybe I can help implement these features for Packit?
Not Packit. We first need to outline the syntax of multihost test definition in tmt. Then implement it in the tmt tool. As the last step we need to make sure this also works in Testing Farm, the service which runs tmt tests for Packit.
Of course, we are very happy for any contribution. So if you have some ideas, how this could work, share your experiences that would be awesome.
[2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests
....
Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do
not
support nested virtualization and we do not yet have bare metal support with AWS
:/.
I see. So a AWS EC2 instance doesn't support starting a VM via qemu, right?
Yes, AWS EC2 does not support nested virtualization in VMs, just on bare metal.
So ... the only way to make this work **now** is to use internal infrastructure, but that is outside of Packit, and you would need to call Testing Farm service yourself (from Github actions for example), scripting the communication. That way, you could, from upstream pull request, provision internal infrastructure and return back results. The only drawback of this solution is that the artifacts storage are not accessible from the public, you will just get an xunit XML with Pass/Fail results, but the links will lead to internal Red Hat infrastructure.
Unfortunately, as we do not yet have any users doing exactly this, this would be a prototype for your team. But should be possible ...
If the internal artifact storage is a no-go for you, we do not have a possible solution right now. But we will be glad to talk about possibilities, like prototyping the use of AWS bare metal instances ....
If the problem is that this goes around Packit, we can have a talk if this feature (run tests against internal infrastucture) is something we could also support via Packit.
This feature is nice to have but I could prefer what has been described before.
Me too :) Proper tmt support would be the best.
Best regards, /M
But this also needs some additional work ...
Unfortunately we are a bit behind with docs, so the best sources we now have are:
https://packit.dev/docs/testing-farm/ https://docs.fedoraproject.org/en-US/ci/tmt/
Thanks for recommending these docs!
But what I wrote here is not included there ...
Best regards, /M
Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you
share
some materials with me since docs.fedoraproject.org/en-US/ci seems to
be
outdated?
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE /
OSCI
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95 TPB-C 2C215 :: Mobile +420 773 944 252 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
-- Best regards, Coiby
On Wed, Apr 14, 2021 at 10:43:26PM +0200, Miroslav Vadkerti wrote:
Hi Coiby,
On Wed, Apr 14, 2021 at 3:57 PM Coiby Xu coxu@redhat.com wrote:
Hi Miro,
[...]
Since Packit supports running tests inside a VM, I expect packet to provision multiple VM instances for a single test and make the console output of the VMs accessible to test.
Unfortunately multihost support is not yet implemented. All testing we do is single host.
So has the feature of VM console output accessible to the test been implemented? If the answer is yes, I want to run the local-kdump test (multihost support is not needed by this test) using tmi to have a first-hand experience of Packit and tmt.
If this works, one great benefit of using Packit is I can run the sames tests for both upstream and downstream kexec-tools and dracut. Because commits backported from upstream dracut could also break kexec-tools.
Yeah, we would like definitely to see multihost supported ...
Btw, the kdump tests have already implemented a framework to provision multiple VM instances and watch the console output. Maybe I can help implement these features for Packit?
Not Packit. We first need to outline the syntax of multihost test definition in tmt. Then implement it in the tmt tool. As the last step we need to make sure this also works in Testing Farm, the service which runs tmt tests for Packit.
Of course, we are very happy for any contribution. So if you have some ideas, how this could work, share your experiences that would be awesome.
I'll share it after getting more familiar with Packit and tmt:)
[2] https://src.fedoraproject.org/rpms/kexec-tools/blob/f34/f/tests
....
Currently, for Fedora CI and Packit, we use AWS EC2 instances, which do
not
support nested virtualization and we do not yet have bare metal support with AWS
:/.
I see. So a AWS EC2 instance doesn't support starting a VM via qemu, right?
Yes, AWS EC2 does not support nested virtualization in VMs, just on bare metal.
I'm curious to know if qemu without kvm can run on on AWS EC2 since QEMU can fall back to software emulation?
So ... the only way to make this work **now** is to use internal infrastructure, but that is outside of Packit, and you would need to call Testing Farm service yourself (from Github actions for example), scripting the communication. That way, you could, from upstream pull request, provision internal infrastructure and return back results. The only drawback of this solution is that the artifacts storage are not accessible from the public, you will just get an xunit XML with Pass/Fail results, but the links will lead to internal Red Hat infrastructure.
Unfortunately, as we do not yet have any users doing exactly this, this would be a prototype for your team. But should be possible ...
If the internal artifact storage is a no-go for you, we do not have a possible solution right now. But we will be glad to talk about possibilities, like prototyping the use of AWS bare metal instances ....
If the problem is that this goes around Packit, we can have a talk if this feature (run tests against internal infrastucture) is something we could also support via Packit.
This feature is nice to have but I could prefer what has been described before.
Me too :) Proper tmt support would be the best.
Best regards, /M
But this also needs some additional work ...
Unfortunately we are a bit behind with docs, so the best sources we now have are:
https://packit.dev/docs/testing-farm/ https://docs.fedoraproject.org/en-US/ci/tmt/
Thanks for recommending these docs!
But what I wrote here is not included there ...
Best regards, /M
Tomas said you are running periodic onboarding sessions. When will you launch the next session? Can you
share
some materials with me since docs.fedoraproject.org/en-US/ci seems to
be
outdated?
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE /
OSCI
IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95 TPB-C 2C215 :: Mobile +420 773 944 252 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
-- Best regards, Coiby
-- Miroslav Vadkerti :: Senior Principal QE :: Testing Farm / BaseOS QE / OSCI IRC mvadkert #tft #baseosci #osci :: GPG 0x7B5B2E95 TPB-C 2C215 :: Mobile +420 773 944 252 Red Hat Czech s.r.o, Purkyňova 115, 612 00, Brno, CR
-
Coiby Xu -
Miroslav Vadkerti -
Petr Šplíchal -
Tomas Tomecek