On Wed, May 04, 2022 at 02:46:16PM +0200, Kamil Paral wrote:
> popularity and reviews noting how polished everything is makes
us very much
> want to build on that. So I understand why this is here, including the
> expanded "all installed applications" Workstation criteria.
As Adam already noted, it is actually cut down. We used to have higher
standards in this area (and we lowered them because we couldn't keep them,
especially when KDE ships a bazillion of preinstalled apps).
Okay, fair. I didn't mean "expanded" in a temporal sense, just the
"expanded
universe" one. But that wasn't clear.
We could have a GNOME Apps Test Day, or perhaps a GNOME Low Profile
Apps
Test Day. Separating that from GNOME DE basics + settings etc would perhaps
motivate people to focus more on those apps and spend more time with them.
I think that would help. More on this coming in my (will write after this
one!) reply to Adam's message.
> 2. "Basic functionality" seems scoped too broadly
currently. I propose,
> for the release criteria, we change this to: A) "the app doesn't
> crash on launch", and B) "the app's behavior does not seem
> immediately embarrassing with a few minutes of playing around with
> it".
If you only want to block on app launch and close, let's be honest about it
and call it that. The basic functionality requirement can stay for those
high-profile apps listed explicitly in the criterion, and the other apps
would only be required to launch and close without crashing.
I would be fine with calling it something else.
However, that's a *massive* step down in quality. Do we really
want that?
Of course not. But....
Your B) is too vague for me. If we ship with our current
photos/contacts/etc bugs, I'll feel embarrassed.
I guess another way to put it is: blocking the release causes huge problems.
It means people don't get updates they were looking for. It messes up the
schedules of people counting on is. It derails publicity momentum. It means
other parts of the project can't get the stuff _they_ have worked so hard on
out to users. And more. Of course, shipping something that's broken *also*
has negative effects (I don't think I need to argue for that, but can if
people want!). Because there are going to be far more things that don't work
perfectly than we possibly have resources to fix, it's always a balance.
So it's always a matter of "which is worse?". Right now, we mostly push
that
out and only (officially) consider it at basically the last minute. I think
that for some things, it'd be better to consider that factor earlier.
Does that make sense? I'm not sure of the best way to do that in practice.
And, I know we've _always_ worked hard on that balance with the "hybrid"
approach to the schedule — I don't mean to discount that.
[things snipped but not because I didn't read them... just trying to not
repeat too much...]
3. Problems found which are not regressions should not be blockers. We're
> just hurting ourselves when we make this our forcing function to get
> something fixed.
Such a broad rule is a really bad idea. You're probably thinking "this bug
was already there, and very few people complained, so why block our next
release on it?". Yes, if we include a safeguard "and very few people
complained", the proposal starts to sound more reasonable. But imagine that
there was a massive disaster in our last release - e.g. a Nautilus
data-corruption bug slipped through our fingers, or Anaconda ate hard
drives in certain cases, stuff like that. We only found out after the
release, when we received a flood of angry bug reports of people leaving
Fedora for good, and by your definition... we are **prevented** from
blocking our next release on that. That's surely not a good idea?
I'm not convinced! If it is at the high end of catastrophic, to reset
ourselves we might do something entirely outside of the normal process (like
when we did the whole-year cycle for F20). And if it's not that bad — but
still important — how does blocking the release for an issue that's already
out there _help_? That data-eating Anaconda would still be on the Get Fedora
page. Rogue Nautilus still out there.
Obviously we should fix it, but why is _blocking_ the tool? What benefit
does it have that other approaches to prioritizing a fix wouldn't?
> trackers in pagure or whereever _or_ having it as more
targets in the
> blockerbugs app. I tend towards the latter: I think it might help
> with the problem where blockers feel like the only obvious way to
> bugs tracked and fixed. But either way, there should be lists!
Well, if there are lists which those teams actually follow and maintain,
that would of course be a good thing. Bugzilla tracker would be the easiest
implementation. I'm a bit worried that those teams will get flooded with an
avalanche of reports, but that's not really my problem to solve. I'm more
than happy to tag e.g. Workstation bugs against the Workstation bugzilla
tracker, to make them aware.
I hope that the teams would be interested. As always, the main problem is
triage. I think the fact that we _do_ have a well-defined (and aggressive)
process for blocker bug triage is one of the reasons a lot of things end up
there.
> 4. Desktop application problems discovered during at the last
minute
> should not be blockers.
[much more snipped -- I think the responses below
cover what I'd say inline
here.]
> If the problem is really going to impact a lot of
> people, it should have been discovered in the beta.
>
But that's the point. The Photos bug is unlikely to affect many people,
because its usage is probably very low. And that's why it wasn't discovered
before. Because testers don't use it regularly, and if you spend one minute
with it, it might seem "OK" to you, depending on which buttons you click. I
wrote my thoughts about this problem in detail here:
https://pagure.io/fedora-workstation/issue/304#comment-795150
There's a _lot_ of software with a _lot_ of bugs. When we block for
not-so-frequently used Photos, it means that other bugfixes and improvements
in other software with more users is held up.
At the same time, I disagree with the intention of this change. Only
Beta-related things get proper testing around Beta, because it's *Beta*,
and therefore we focus on *Beta* stuff. Even then, we often miss
Beta-blocking bugs and discover them before Final. We can't test (not
deeply, and sometimes at all) Final-related stuff, because we simply don't
have time for it during Beta. Not to mention that Beta-related bugs
sometimes preclude actually testing Final-related things. We can only start
properly testing Final once Beta is out. Also, let's not forget that GNOME
completely changes around Beta with a new major update. I simply don't see
how this "should have been discovered in the beta" could happen in the real
world.
There's a tension, right? At one end, the rough-to-polish direction that
we're going towards the release. But coming from the other side: less and
less time to fix problems as we get closer. It's hard to reconcile. Pushing
the release back really only creates the _illusion_ of more time (unless
we're willing to go back to slipping the schedule constantly around the
calendar, which I'm really really sure we should not).
> (Exception for _new_ regressions, of course.)
> I don't know how to phrase this in a way that doesn't make Adam sad with
> me, but maybe something like: Desktop application blockers discovered
> during the final freeze are automatically waived
I'm sad as well :sad panda:. This would effectively say "we don't care
about Desktop". We would ship it horribly broken. Again, if Nautilus eats
your documents, it's not a blocker, just because it was discovered after
Beta. Why such a broad statement? And why do you single out desktop apps in
particular? And why all of them instead of some subset, like the
low-profile ones?
What we care about and what we can do are different. I care about desktop.
But anyway, you split off a crucial part of my suggestion:
Desktop application blockers discovered during the final freeze are
automatically waived _unless the relevant Spin or Edition team decides
otherwise_.
There's a handle to pull if escalating is warranted.
And now a bit more technical. How do you want to decide if the bug
was
present before Beta, but discovered after Beta, or if it appeared after
Beta? The nightly composes are only stored on our servers for 2 weeks (!).
I probably shouldn't use the US Patent system as a basis for anything,
but... what the heck. This is why that changed to date of filing. No
complicated proof, just date proposed as a blocker. (And again, if it turns
out to have been discovered earlier and _should_ be escalated, it could be.)
[...]
unless the relevant Spin
> or Edition team decides otherwise.
>
All the power to the working groups. I'd happily let them decide about all
bugs related to their Edition, and maintain their own release criteria and
everything, because that would be way less work for QA :-) But I don't
suppose they'd jump in joy about this.
I'd like to push more work there, yeah, and have QA covering the base, the
tooling and processes and standards, and acting as consultants. But also,
the original plan (which I still believe in!) was to make sure there was a
QA team representative on every working group, and presumably that person
would be connected with both the WG and the team on decisions like this.
So unless they really want all this responsibility, perhaps it could
work
the way around - accepted blockers could be waived by the decision of the
relevant team. It's their product after all. Of course, there should be
some systematic approach, so that we don't waste much time discussing
blockers which are then getting waived. And if there are many waivers in a
certain area, the related criteria should be adjusted to reflect that. But
in general, I think this approach could work fine.
There are two reasons I like this the direction I proposed ('fail open'
rather than 'fail safe').
First, it moves the decision earlier; right now, waiving a blocker is very
late in the process — that means if it's not waived, it's definitely too
late to do anything about it. (Which might particularly be an issue if the
people who could implement a fix assume that it will be waived.)
More crucially, it moves the default _pressure_ from "this will ruin the
schedule from everyone else so you better fix it" to "oh no! there's a bug
in our Edition and we should fix it before the release is out".
> 5. Okay, and... bigger: we should aim for more approaches which
let us
> decouple as much as possible from the Release. (My grand hope is that we
> can release every deliverable on its own schedule, but I also understand
> the _highly aspirational_ nature of that idea. But...) What if we could
> just easily ship GNOME Photos from GNOME 41 until a fix is found in the
> updated one?
The answer could be Flatpak. And honestly in our criteria there's nothing
preventing Workstation WG doing just that right now. But technically we're
not ready yet, I guess. If this can be done with RPMs as well, let's go
ahead. Ubuntu does that all the time.
I don't think we can easily do it at the rpm-alone level. Too much is
integrated. [Mumbles something incomprehensible about modularity, stares
into distance.] But I think Flatpak is definitely worth exploring to help
here (plus OCI/Podman containers for non-desktop apps).
After reading all these proposed changes, I have to ask - what is the
main
motive for proposing them? Is it to release F36 already? Is it to prevent
future Fedoras from delaying as much as F36? We used to be OK with release
slips. Is desktop importance lower than before? Is it the frustration from
finding trivial bugs in trivial apps so close to the final release?
Something else?
In order: too late for that; yes; I know, and it was bad; absolutely not;
yes; and: generally looking at how to improve things both the Edition teams
(particularly desktop, obviously), QA, the release as a whole including
other Editions and spins impacted, and users.
Because depending on what we want to achieve, perhaps there is a
better
way. With the current proposed changes, I believe that the end result would
be less-delayed Fedora with more broken desktop apps. The reason for
introducing the basic functionality criterion in the past was, iirc, a bad
PR we were given in reviews when desktop apps often broke quickly after the
reviewer tried to use them. It seems we'd be heading back in that direction
with this proposal.
Instead of shipping broken apps, what if we had a conversation about which
is better - shipping broken apps or not shipping those apps at all? And I
mean this question honestly. Is it better to ship something we *know* it's
Do you mean disable the app so it isn't included in the repo? (If so, what
does that do to people who upgrade?) Or at least hidden from Software?
It makes sense to scope the release critera to "on the media", because we
have to draw a line somewhere. But shuffling things across the line by
moving them from the default install doesn't _really_ improve quality from a
user perspective. I don't have Photos installed on this system, but if I
type "photo" in the GNOME Shell search, I'm given the suggestion to install
it.
in a broken state, and hopefully issue an update later, or is it
better to
yank it from the default install (provided it's not crucial for the
desktop)? Or delay the release? I'd start with defining our priorities in
this way. Is it unthinkable to have a plan like "apps from group X can only
delay the release for at most Y weeks, otherwise we'll not ship them by
default"? It would also make us re-evaluate whether we really need to ship
everything we currently have, including unmaintained apps without any
developers, or half-baked apps with just a slight community maintenance,
etc. I don't mean to be derogatory to some of those gnome apps. But if
those apps are problematic, is the right approach to lower the quality bar
for all apps included, or should we rather make some adjustments just for
the problematic set?
I do like in general where you're going with this. This is one of the things
I wanted to do with the "Rings" proposal — make it more clear what we're
able to focus on and what we're not. All of that other software could be
available in some way, but clearly separate.
--
Matthew Miller
<mattdm(a)fedoraproject.org>
Fedora Project Leader