On Tue, Aug 25, 2020 at 7:25 PM Dusty Mabe <dusty(a)dustymabe.com> wrote:
On 8/21/20 8:06 AM, Kamil Paral wrote:
> Hello folks,
>
Hey Kamil!
Hey Dusty, sorry for a late reply.
> wanted to thank you all for supporting the FCOS Test Day which turned
out to be a very successful event! As we looked back, we thought there
might be room for more community participation in FCOS QA. On those lines,
we were wondering if there are good ways for other Fedora teams and for the
community in large to get a good picture of how quality validation and
decisions are done for the FCOS stable stream, and how people who are
interested can participate in this process.
>
> In particular, I tried to find some documentation on:
> * The pool of bugs which are currently reported against the testing and
next stream
> * The subset of aforementioned bugs which were decided to be "blockers"
and must be fixed before the stream can be promoted to stable
> * Where this blocker decision-making takes place and how people can get
involved
> * What are the procedures and rules for this blocker decision-making
process
>
> Unfortunately, I was not able to find any. I might be looking at a wrong
location. Can you please point me to the correct place, if such
documentation exists?
Our high level bugs mostly exist in the issue tracker. You are right we
could use more transparency in our process.
https://github.com/coreos/fedora-coreos-tracker/issues
So if there is an important issue in bugzilla, do you create a copy of that
issue in your tracker? Or what do you do?
Do you have some labels or other ways to specify which bugs are blocking
the next stable update? How do you track whether the current testing stream
is good to go?
>
> Since FCOS might soon be an official Fedora edition, it would be great
to have insight into the FCOS community release process. But this is far
from just a QA team thing. This process transparency is important for
recruiting new passionate community members who can help you run and
improve things. Another use case might be a party affected by a bug who
might feel motivated to visit such a blocker discussion and argue their
case about why they think such a bug should not hit the stable stream.
+1. That's a good point. Obviously our release process happens more often
and we don't want to add a lot of
manual "process" around it because then it gets hard to manage, but I
think we should improve.
I understand that. But you still need to manually inspect bugs and figure
out which are critical ones, right? That's the process I'm talking about.
How do you currently decide which ones are critical? During a meeting? Do
you have some ways to partially automate this decision (I can imagine you
could e.g. pinpoint a build which makes your testsuite fail and
automatically create a ticket for it)?
>
> We can put together all the relevant documents in one place that is
accessible to the public so that the community is aware of important
decisions made during the release process and feel included. I imagine that
there could be a "Participate!" section in the FCOS docs page [1] which
would point people to areas where they can help FCOS, and one of those
areas could be quality control. What do you think?
I think the most organic way people can get this information into the
community today is just open a tracker issue
and come to our meeting.
Sounds good. That's simple enough, but serves the purpose well. It just
needs to be visibly stated somewhere.
For example some people raised issues with the new version of podman
and
we're currently
holding it back in FCOS. See the following two issues:
-
https://github.com/coreos/fedora-coreos-tracker/issues/560
-
https://github.com/coreos/fedora-coreos-tracker/issues/575
>
> As an example, the current quality validation workflow in Fedora
includes a regular "blocker review" meeting in a dedicated IRC channel,
where bugs proposed as "blockers" get discussed [2][3]. Anyone can propose
a bug as a blocker and we regularly receive bug nominations from the
community. The blocker review meeting is open to anybody and we try to base
our votes on "release criteria" [4] after assessing that particular case
and its circumstances. Release criteria help us bring a bit more
consistency into this blocker decision-making and it saves us some time
from looking up historical evidence ("how did we vote last time on a
similar bug?"). All currently proposed and accepted blockers are visible
from our custom page [5]. This whole process is a collaborative effort
between Release Engineering, Quality Assurance, Development, and Project
Management. Please note that while I'm describing our current process as an
example, I'm not implying you need to do it in the same way.
> I'm just showing how we do it in general. Also, our documentation is
quite complex and lengthy, but essentially it can be just a couple of
paragraphs with links, to steer people into the right direction.
>
> Looking forward to hearing your thoughts. Thanks!
The existing process for fedora is well thought out but maybe a little
overwhelming for us for right now. Clement, who
recently joined our team, is going to try to help us with this in the
future.
Perfect. I'll be happy to give him some advice, if he's interested.
In the mean time, I think the best way to go about making change here
would be for some concrete proposals for how we
could implement some baby steps to increase transparency. Kamil, do you
have any suggestions there?
Sure. So let's talk about visibility first. If I open your documentation
[1] I see a link to your IRC and Discourse and then in the FAQ I found this
list of links [2]. Quite interestingly, the link to the
fedora-coreos-tracker github is missing. I also haven't found any mention
of reporting bugs nor your meetings. I found the tracker mentioned in
Getting Started [3], but I would expect it in the FAQ "communicate" section
(or I'd expect a follow-up "reporting issues" section). So perhaps the
first step could be to make sure the tracker is visibly mentioned among the
communication means. As a bonus, you can then avoid duplicating that
information in other places (like Getting Started) and just link to that
section from everywhere. That "canonical location" doesn't even need to be
in the FAQ, but can be in your tracker [4] and the FAQ section can link
there as well. It depends how much you want to deduplicate information. It
helps to make sure some channels are not forgotten in one place, and other
channels in a different place.
A second step could be to create some kind of Participate/Join us section.
There you could list ways how people can help improve FCOS and become part
of the community. I'm sure it's not just about QA. But regarding QA, there
could be things like reporting issues, going through the existing issues
and trying to reproduce them on their end, supplying more
information/debugging the problems, and also participating in the meetings.
Currently your Meetings section [5] is more of a guide on how to run
meetings, and the Voting section below contains rules, but little
information on what topics you actually vote. It might help to extend the
Meetings section to state that it is open to the public, that you'll be
happy to see new faces, and what is the usual agenda on such meetings.
Similarly for voting, it could say what topics you usually vote on and that
you'll be happy to hear opinions from the public before the vote. You could
move some of the technical info ("Steps to run the meeting") into a
separate document and link it, to increase readability.
Does that sound reasonable, is it helpful?
Most of this has been about community participation, but there is a bigger
picture. For example, understanding your processes and being able to
quickly assess the current state of the in-development version of FCOS is
very helpful even for QA or release management. It's all linked.
Thanks,
Kamil
[1]
https://docs.fedoraproject.org/en-US/fedora-coreos/
[2]
https://docs.fedoraproject.org/en-US/fedora-coreos/faq/#_what_are_the_com...
[3]
https://docs.fedoraproject.org/en-US/fedora-coreos/getting-started/#_gett...
[4]
https://github.com/coreos/fedora-coreos-tracker#communication-channels-fo...
[5]
https://github.com/coreos/fedora-coreos-tracker#meetings