On Mon, Feb 17, 2020 at 3:20 PM Kamil Paral <kparal(a)redhat.com>
wrote:
> This proposal intends to reduce the scope of the “*Default application
> functionality*” release criterion [0]:
>
> All applications that can be launched using the standard graphical
>> mechanism of a release-blocking desktop after a default installation of
>> that desktop must start successfully and withstand a basic functionality
>> test.
>>
>
> *= Background =*
>
> The area which QA is responsible for has been growing and growing in
> recent years. We used to have just a single Fedora release with two
> blocking desktops (GNOME and KDE). Then Editions got introduced and we
> started testing Workstation, Server, Cloud and the KDE Spin. Additional
> architectures were introduced (fortunately i386 got obsolete) and we
> started testing and blocking on specific images on armhfp and aarch64.
> Currently there are 13 release blocking deliverables mentioned on the
> ReleaseBlocking page [1]. That list is not complete, though. Fedora CoreOS
> became an official edition lately, and even though its release cycle is not
> tied to traditional Fedora release cycle (and that’s why it’s not mentioned
> on that wiki page), as an official edition QA should care about it as well.
> It is the same story with Fedora IoT, another recent release-blocking
> addition [2]. Desktop testing is one of the most time-consuming jobs with
> the most frequent bug occurrence. Right now, we’re supposed to be fully
> testing and blocking on GNOME, KDE and XFCE (although XFCE might get
> dropped in favor or aarch64 Workstation [3]). We cannot test all of this
> and honestly claim that we verified basic functionality of all the included
> apps on all these desktops. I believe we need to adjust the criterion and
> align it closer with reality. In my eyes, it’s better to have a narrower
> scope and perform it well than having a large scope and perform it poorly.
>
> *= Proposal =*
>
> Change the criterion to something along these lines:
>
> All applications that can be launched using the standard graphical
>> mechanism after a default installation of Fedora Workstation on x86_64
>> architecture must start successfully and withstand a basic functionality
>> test.
>>
>> For other release-blocking desktops (on any architecture), the
>> requirements only apply to the following types of applications:
>>
>> * web browser (e.g. firefox) [4]
>>
>> * file browser (e.g. nautilus)
>>
>> * package manager (e.g. gnome-software)
>>
>> * image viewer (e.g. eog)
>>
>> * document viewer (e.g. evince)
>>
>> * text editor (e.g. gedit)
>>
>> * archive manager (e.g. file-roller)
>>
>> * terminal emulator (e.g. gnome-terminal) [4]
>>
>> * problem reporter (e.g. abrt)
>>
>> If there are multiple applications of the same type (e.g. several web
>> browsers), only one of them needs to satisfy the requirements.
>>
>
> As you can see, the original criterion was kept for Fedora’s flagship
> desktop edition, the one that is most prominent on
https://getfedora.org
> and probably the one that most newcomers download. We would still verify
> everything on Fedora Workstation on x86_64. But any other desktop
> (including Workstation on aarch64) would get just reduced criteria, because
> we simply can’t ensure the same quality bar for the smaller desktop
> editions/spins. There are some high-profile types of applications that I
> considered including in the list above, but didn’t in the end:
>
> * word editor (e.g. libreoffice-writer)
>
> * spreadsheet editor (e.g. libreoffice-calc)
>
> * video player (e.g. totem)
>
> * help viewer (e.g. yelp)
>
> I’d like to hear your thoughts on whether they should be included or not.
> Of course from an end-user point of view, it would be beneficial. But the
> question is whether we as QA can promise their testing. And also whether we
> want to block the release e.g. if Gnumeric is broken on armhfp XFCE or if
> totem doesn’t work on aarch64. Yes, it’s unpleasant, but people using
> alternative desktops and architectures are usually far from beginners. It’s
> usually not difficult to install a different application. Also note that
> the apps included in x86_64 Workstation will be thoroughly tested so they
> should be very likely to work well even on other architectures (minus some
> arch-specific issues).
>
> I’d like to target Fedora 32 with this proposal, which means we should
> decide on this proposal fast. (I wanted to propose it much sooner, but I
> was waiting on clarifications about new release-blocking images and also on
> some fesco tickets [3], some of which is still not clarified; so I’m sorry
> about a late proposal).
>
> Please comment, thanks.
>
>
> [0]
>
https://fedoraproject.org/wiki/Fedora_32_Final_Release_Criteria#Default_a...
>
> [1]
https://fedoraproject.org/wiki/Releases/32/ReleaseBlocking
>
> [2]
https://pagure.io/fedora-iot/issue/23
>
> [3]
https://pagure.io/fesco/issue/2339
>
> [4] These two are required to work even in Basic release criteria [5],
> but I decided to include them anyway to avoid confusion (“why is the web
> browser missing?”), and to make it clear that they need to satisfy the
> basic functionality (whereas for Basic release criteria just the barebone
> functionality might be deemed enough)
>
> [5]
>
https://fedoraproject.org/wiki/Basic_Release_Criteria#Required_applications
>
>
>
I haven't received much feedback on this proposal. On test list and kde
list, there was a short follow-up regarding the list of release-blocking
applications. On kde list, Kevin disagreed because it lowers the guaranteed
quality bar for KDE, but he didn't continue discussing it with me. I
haven't received any alternative proposals either. On desktop list, there
was silence, but Chris Murphy gave me a thumbs up on IRC.
I'll give it a few more days, and unless somebody yells hard (and ideally
also provides some constructive ways how to optimize desktop testing,
because we need to cut it down /somehow/), I'll put the change into effect.
I'll also probably move the help viewer into the list of release-blocking
applications, based on the discussion I saw.
Hi Kamil,
I think this change makes sense. +1 from me as well.
Kalev