Hello everyone,

on behalf of the Fedora QA team I'd like inform you about some useful QA processes and links that can help you with your Fedora CoreOS releases. Here we go.


= Test Days =

When you have something new and interesting, or perhaps something you'd like to get tested by different people/workflows/hardware configurations, it's a good idea to ask the community to participate in a test day. You provide the thing to be tested (an image, package, etc) and the test cases, and we'll try to spread the word and support attendees.

You can see the schedule, the instructions and other important info at:
https://fedoraproject.org/wiki/QA/Test_Days


= Blockers and freeze exceptions =

Blocker process applies to products that are blocking Fedora release (which is not CoreOS's case at this moment), but the freeze exception process can be used by anyone. Before the major release milestones (Beta, Final), the whole Fedora release goes into a freeze (see schedule [1]). Nothing can enter stable updates except for builds that either fix an accepted blocker bug, or have been granted a freeze exception.

When you have an update that you really need to get to stable updates, you can ask for an exception. You explain why you need it, and during a blocker bug meeting the importance of your use case will be weighted against the risk of breaking something in our release-blocking products. A freeze exception is then granted or denied, and the package is pushed stable (or stays in testing until the freeze is over).

The process is described in detail at:
https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process

We have a webapp that can be used to propose the freeze exception:
https://qa.fedoraproject.org/blockerbugs/  (the Propose button)
or it can be done manually in bugzilla (see the SOP).

[1] https://fedoraproject.org/wiki/Releases/29/Schedule


= Common Bugs =

When there's an important or perhaps just an uncomfortable bug in your product, it's good to let your users know. One of the more visible locations that gets referenced in release announcements is the "common bugs" wiki page. It's used both pre-release (important bugs we're trying to fix) and post-release (bugs we failed to fix and were not critical enough to block the release, or bugs we found out too late).

F29 common bugs will be documented at (wiki page not yet created):
https://fedoraproject.org/wiki/Common_F29_bugs

Here are F28 common bugs as an example:
https://fedoraproject.org/wiki/Common_F28_bugs

The process of including a new entry is documented on the same wiki page. You don't need anyone's permission, simply follow the process and edit the page. Alternatively, you can mark it in bugzilla, and someone from QA will get to it eventually and document it for you.

Usually we try to provide a short and understandable description of the bug on that wiki page, so that even end users without deep technical knowledge can read and understand it, and document a workaround or some other solution when possible.


= Release validation matrices =

We have a process of using wiki matrices for tracking release validation testing. It contains the test cases and their results for a particular compose. The results are partly provided from manual testing and partly from automated systems. Here's an example of F28 final compose summary page (containing all matrices grouped into a single view):
https://fedoraproject.org/wiki/Test_Results:Fedora_28_RC_1.1_Summary

As I learned, you give a heavy emphasis on automated testing and don't intend to have manual test cases at the moment. Therefore we probably won't extend our matrices with FCOS test cases and results (at least not yet, might change in the future, e.g. if it changed to be release-blocking). All you need to do is to pay attention to your automated test suite results. It would be nice if Fedora QA had access to those results as well, and we've been discussing that on your github issue tracker. One of the options is to make your test runner public, another (possibly combined) is to send at least the overall results to our ResultsDB instance [1], so that it's stored in the same place as the rest of our automation results and we can check on it in a programmatic way.

So, this might not be immediately useful to you, but we'll try to work on it in the future.

[1] https://taskotron.fedoraproject.org/resultsdb/


= Communication =

I'm not sure whether I included all that is relevant or forgot something. Either way, if you have any questions or requests, please contact us. You can either ask here or on IRC (I and coremodule will be following your channels), or you can use our QA channels:
https://fedoraproject.org/wiki/QA#Communicate


Thanks,
Kamil
Fedora QA