I worked with FPL Matthew Miller and our engineering manager Jim Perrin, among others, to define the various problems we want to solve in diversifying the Fedora lifecycle. We're seeking review and feedback from community members. The most salient feedback will be from those involved in the efforts we describe, but we welcome all constructive feedback.
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
* * * Fedora’s singular lifecycle has been in place for almost a decade and a half. During that time, technology users have changed how they consume platform and applications. Fedora needs to be more toward the forefront of these changes. But more importantly, Fedora needs to be more hospitable to community management of lifecycle.
Currently Fedora can’t scale for more community ownership of the things we release: (1) Only a few people can build and push out releases; and (2) we manage releases largely based on that staffing. The Fedora community should be able to run releases of content themselves, using tools that work well, with only minimal oversight, and determine their own schedule for doing so.
This implies a great deal of both redesign and reworking of tools and processes. To unblock the community, several things need to happen. We need a faster, more scalable compose to enable CI/CD operations; we need to automate more testing and quality measures; and we need to update our delivery tools and processes. We also need to track and coordinate this work across teams, since it involves collaboration among Fedora infrastructure, QA, applications, release engineering, CentOS CI, maintainers, and more.
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency. * * *
The full page is here, with a set of problems, solutions, and actions proposed. I invite you to take time to read it in detail: https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements
I also attached that page to the main Lifecycle objective page here, which was previously approved by the Council: https://fedoraproject.org/wiki/Objectives/Lifecycle
Rather than try to spin this one email into a thread of doom that people will give up on reading, I encourage those with feedback to open a thread for any particular topic. That way the community discussion should be more useful for all.
I'll collect input and use it both for responses and to help tweak plans for (hopefully) optimal results. I'm on vacation next week, for which timing I apologize. But I'll try to look at mail here from time to time and when I return.
-- Paul
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
* embrace Taiga (an open source kanban tool) for project planning * fix the compose speed (target: one hour!) * really actually for real gated Rawhide * better CI pipeline tests for everything * define a base platform -- Red Hat wants to focus resources here * better tooling for non-base deliverables * better metrics for everything
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements and comment in this thread.
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
I am +100 to these changes.
The idea that we can essentially not release for a year to do this raises the interesting point that perhaps our release cadence is not as rigid as we think it may be. I don't think we are going to skip Gnome and other major updates, are we? If not, then this also runs into the conversation about longer lifecycles and whether we need releases, imho.
regards,
bex
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements and comment in this thread.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Nov 26, 2018 at 4:23 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
I am +100 to these changes.
The idea that we can essentially not release for a year to do this raises the interesting point that perhaps our release cadence is not as rigid as we think it may be. I don't think we are going to skip Gnome and other major updates, are we? If not, then this also runs into the conversation about longer lifecycles and whether we need releases, imho.
I like some of the ideas here but I feel we need to have an extremely well defined set of problems to be solved else we'll delay the release, achieve very little, and either never release again or not have anything achieved except a waste of time.
On 11/26/18 8:21 AM, Brian (bex) Exelbierd wrote:
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
I am +100 to these changes.
The idea that we can essentially not release for a year to do this raises the interesting point that perhaps our release cadence is not as rigid as we think it may be. I don't think we are going to skip Gnome and other major updates, are we? If not, then this also runs into the conversation about longer lifecycles and whether we need releases, imho.
Indeed.
If we push as updates things we normally wait for releases for, would that disrupt our users?
Is there a cutoff of things we would push in updates? For example, we usually do mass rebuilds of the entire collection of packages when new gcc/etc are ready, would we do that as a update in a stable release?
We could solve the rolling changes issue by requiring all major changes to be modular and not enable the new module (ie, gnome 3.32 releases in f30, in 6 mmonths 3.34 comes out, we enable it as a new module, users can choose to switch to if they want, or stay on 3.32 if they ignore it or don't want to right then). That won't help for tool changes tho.
kevin
On Mon, Nov 26, 2018 at 5:14 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
That would be good, however I don't know how it is related to lifecycle of distro.
P.S. would be great if someone finishes packaging of it (I did a lot of work here, but not everything).
- fix the compose speed (target: one hour!)
Do we have analysis of what is taking how much time?
- really actually for real gated Rawhide
Is the "creating side tag for chain builds or by automated requests" is planned here? When I deal with rust packages, I often need to build some update which will break other packages and I need to fix them, but if something will prevent going that build in buildroot.
- better CI pipeline tests for everything
Does it mean that finally someone will make it usable? Do we have definition of "better"? I've tried to use CI for all rust-* packages, but it is basically unusable.
* No way to re-trigger tests * Koji repo is broken even is enabled (https://pagure.io/fedora-ci/general/issue/14, no single comment for > 0.5 month) * Doesn't re-trigger after dependency changes * No way to tell that set of tests should be run on some file pattern (copy-cating same tests folder to 480 packages is not much fun)
- define a base platform -- Red Hat wants to focus resources here
Can you elaborate?
- better tooling for non-base deliverables
Are Ursa Major and Freshmaker included in here?
- better metrics for everything
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements and comment in this thread.
I would appreciate having this thread on discourse because since introduction it is my main place where I try to have such conversations.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote:
- really actually for real gated Rawhide
Is the "creating side tag for chain builds or by automated requests" is planned here? When I deal with rust packages, I often need to build some update which will break other packages and I need to fix them, but if something will prevent going that build in buildroot.
Hey Igor!
Indeed, I do plan to have Bodhi manage side tags in order to accomplish this. There's a Bodhi kanban board that covers the various items planned to be part of this:
https://github.com/fedora-infra/bodhi/projects/3
You can see that a few of the cards in there relate to side tags. My current thinking is that you will be able to create a side tag update, which will work slightly different than a normal update in a few ways:
* You create the side tag update before you make any builds, instead of after. * The side tag update makes, well, a side tag in Koji for you. * You go build all the builds you need in your side tag. * When you are done, you tell Bodhi you are ready to go to testing, and it will work like a normal update from there on.
Does that sound useful to you?
On Mon, Nov 26, 2018, 20:09 Randy Barlow <bowlofeggs@fedoraproject.org wrote:
On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote:
- really actually for real gated Rawhide
Is the "creating side tag for chain builds or by automated requests" is planned here? When I deal with rust packages, I often need to build some update which will break other packages and I need to fix them, but if something will prevent going that build in buildroot.
Hey Igor!
Hey Randy,
Indeed, I do plan to have Bodhi manage side tags in order to accomplish
this. There's a Bodhi kanban board that covers the various items planned to be part of this:
https://github.com/fedora-infra/bodhi/projects/3
You can see that a few of the cards in there relate to side tags. My current thinking is that you will be able to create a side tag update, which will work slightly different than a normal update in a few ways:
- You create the side tag update before you make any builds, instead of after.
- The side tag update makes, well, a side tag in Koji for you.
- You go build all the builds you need in your side tag.
- When you are done, you tell Bodhi you are ready to go to testing, and it will work like a normal update from there on.
Does that sound useful to you?
It definitely does, just wanted to ensure if it is one of things which is supposed to be worked on if we skip release.
_______________________________________________
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Dne 26. 11. 18 v 20:01 Randy Barlow napsal(a):
On Mon, 2018-11-26 at 17:23 +0100, Igor Gnatenko wrote:
- really actually for real gated Rawhide
Is the "creating side tag for chain builds or by automated requests" is planned here? When I deal with rust packages, I often need to build some update which will break other packages and I need to fix them, but if something will prevent going that build in buildroot.
Hey Igor!
Indeed, I do plan to have Bodhi manage side tags in order to accomplish this.
Does that mean that Bodhi will be finally enabled for Rawhide or not? Because otherwise I can't see how Bodhi is related here.
Vít
There's a Bodhi kanban board that covers the various items planned to be part of this:
https://github.com/fedora-infra/bodhi/projects/3
You can see that a few of the cards in there relate to side tags. My current thinking is that you will be able to create a side tag update, which will work slightly different than a normal update in a few ways:
- You create the side tag update before you make any builds, instead of after.
- The side tag update makes, well, a side tag in Koji for you.
- You go build all the builds you need in your side tag.
- When you are done, you tell Bodhi you are ready to go to testing, and it will work like a normal update from there on.
Does that sound useful to you?
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, 2018-11-27 at 11:56 +0100, Vít Ondruch wrote:
Does that mean that Bodhi will be finally enabled for Rawhide or not? Because otherwise I can't see how Bodhi is related here.
Indeed, the plan is to make Bodhi manage Rawhide as well.
On Mon, Nov 26, 2018 at 10:03 AM, Matthew Miller mattdm@fedoraproject.org wrote:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
Can we use this as an opportunity to work out how to use the blocker bugs process for a megaupdate? Our reputation for providing an up-to-date desktop took a hit last time we did this, so we'll want to provide a GNOME 3.34 megaupdate in October 2019 to compensate for the missing Fedora release there. We'll need a monthslong testing process for this to go smoothly mid-release.
Michael
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis. The fact of the matter is that the compose has a LOT of I/O and I/O is slow and the cost to upgrade the infrastructure to get high through put I/O is expensive and no one has ever offered the budget to resolve that problem. The fact of the matter with this is that we now produce a lot of release artifacts and that takes time, some of it can probably be run more in parallel, and possibly some things run at different times but without a combination of investment in infrastructure, and maybe even hacking and slashing of components this is not simply a check box.
For example the IoT compose takes around an hour but ATM we currently produce 5 artifacts, mostly in parallel, and don't do things like newRepo for the base rpms because we consume that from the main compose.
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
They are very nice top level ideas, but each of those components need to have in depth analysis and not just the hand wavy overview in the link below.
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements and comment in this thread.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
The fact of the matter is that the compose has a LOT of I/O and I/O is slow and the cost to upgrade the infrastructure to get high through put I/O is expensive and no one has ever offered the budget to resolve that problem. The fact of the matter with this is that we now produce a lot of release artifacts and that takes time, some of it can probably be run more in parallel, and possibly some things run at different times but without a combination of investment in infrastructure, and maybe even hacking and slashing of components this is not simply a check box.
That's why the proposal is to skip an entire release to give us time to work on it. If it was easy, that wouldn't be necessary.
There are of course at least *two* possible solutions to "there's lots of I/O". One is "throw money at making the I/O faster", the other is "figure out how to do less I/O".
One hour is a *target*, not a *requirement*. I don't think anyone's going to consider the time wasted if we wind up getting it down to two hours instead of one.
(Though personally I'd like ten minutes. And two unicorns.)
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
The fact of the matter is that the compose has a LOT of I/O and I/O is slow and the cost to upgrade the infrastructure to get high through put I/O is expensive and no one has ever offered the budget to resolve that problem. The fact of the matter with this is that we now produce a lot of release artifacts and that takes time, some of it can probably be run more in parallel, and possibly some things run at different times but without a combination of investment in infrastructure, and maybe even hacking and slashing of components this is not simply a check box.
That's why the proposal is to skip an entire release to give us time to work on it. If it was easy, that wouldn't be necessary.
There are of course at least *two* possible solutions to "there's lots of I/O". One is "throw money at making the I/O faster", the other is "figure out how to do less I/O".
One hour is a *target*, not a *requirement*. I don't think anyone's going to consider the time wasted if we wind up getting it down to two hours instead of one.
(Though personally I'd like ten minutes. And two unicorns.)
Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
* The mock fsync change (https://pagure.io/releng/issue/7909) may help some.
* We have plans to redo the setup on the s390x builders, which should result in them being much faster. That should help.
* I know pungi maintaienrs have been doing some work to make things faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
kevin
On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote:
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be.
On Mon, Nov 26, 2018 at 7:42 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote:
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > Here's the summary from the page, which proposes we pause the release > after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be.
Yes, that was one of the ideas I had. Basically do the "newRepo" part of the compose that generates the new set of tagged in and signed rpms and base network isntallers 3-4 times a day and then run all the other components separately (live/arm images, CoreOS, IoT, cloud images) on their own schedule in separate components consuming the updates. This is basically what some of the bits do now as you'd indicated. It then allows focus to be put on each component to speed each piece up independently rather than as a whole.
Peter
On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote:
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller <
mattdm@fedoraproject.org> wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > Here's the summary from the page, which proposes we pause the
release
> after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I
expected a little
more interest in this post. Perhaps it seemed like too much text
to digest
along with turkey and stuffing. :) I'm highlighting it with a
subject
reflecting the big, direct impact, and here's some other
top-level
proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be.
This would be great.
I want to get to a state where different WG's can run their own composes by calling a service.
They might actually release on their own schedules as well. (Not all, but something like Labs or Spins) .
-- Adam Williamson Fedora QA Community Monkey IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net http://www.happyassassin.net _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 11:30 -0800, Kevin Fenzi wrote:
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be.
Adam puts his finger on an important aspect of the work, which is to "decompose the compose" (sorry). We need the ability to produce enough A, B, C and D (abusing his example) to do integration tests for specific inputs, such that maintainers get feedback in a reasonable timeframe on their builds. With that, gating Rawhide can be an actual thing.
On 11/26/18 11:55 AM, Paul Frields wrote:
On Mon, Nov 26, 2018 at 2:42 PM Adam Williamson
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
This hasn't been true for a while (we actually have something like half a dozen or more different "composes", right now), but the system is still less flexible and it's still more difficult to configure different composes than it really ought to be.
Agreed, that would be nice. The one issue I see off hand is that koji tags are kind of expensive so we can't just tag everything the way we may want to, but perhaps we can come up with some clever workflow for that.
Adam puts his finger on an important aspect of the work, which is to "decompose the compose" (sorry). We need the ability to produce enough A, B, C and D (abusing his example) to do integration tests for specific inputs, such that maintainers get feedback in a reasonable timeframe on their builds.
Yeah, thats long been a desire of releng/qa... weeding out the obviously broken would be very nice.
With that, gating Rawhide can be an actual thing.
Indeed.
kevin
On 11/26/18 3:53 PM, Kevin Fenzi wrote:
The one issue I see off hand is that koji tags are kind of expensive so we can't just tag everything the way we may want
Can you please elaborate a bit on this? What makes them expensive?
On Tue, Nov 27, 2018 at 9:08 AM John Florian jflorian@doubledog.org wrote:
On 11/26/18 3:53 PM, Kevin Fenzi wrote:
The one issue I see off hand is that koji tags are kind of expensive so we can't just tag everything the way we may want
Can you please elaborate a bit on this? What makes them expensive?
Koji tags are representations of full repositories. They work more or less the same way that Copr projects do, it's just a lot less obvious because of the way Koji represents them in the various UIs.
On Mon, Nov 26, 2018 at 11:39:52AM -0800, Adam Williamson wrote:
While we're on the topic of unicorns, if I were the person actually in charge of this and had the freedom to do it how I wanted, what I'd really want to do is create a much more *modular* compose process, where inputs and outputs were defined and associated with each other, and you could easily define subsets to be produced at different times for different reasons. Not a compose, but a Compose Construction Kit. A big chunk of the length of "the compose" is associated with the fact that we define "the compose" as a single thing which produces (last time I counted) 70+ deliverables. If we had an easy way to say 'ok, right now we need a compose which produces A, B, C and D from the minimum necessary sets of inputs', that would be a massive improvement.
+1000
On Mon, Nov 26, 2018 at 2:30 PM Kevin Fenzi kevin@scrye.com wrote:
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller <
mattdm@fedoraproject.org> wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a
little
more interest in this post. Perhaps it seemed like too much text to
digest
along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
I totally agree, also as Peter mentioned, IOT composes are just taking 1 hour as its just consuming the repo rather than generating newRepo as the normal nightly composes does. If we can generate the newRepo on the fly whenever a package gets build and just use the repo to generate the artifacts either individually (splitting the compose) or parallely (as we are doing right now mostly) that would really help.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Nov 26, 2018 at 7:47 PM Mohan Boddu mboddu@bhujji.com wrote:
On Mon, Nov 26, 2018 at 2:30 PM Kevin Fenzi kevin@scrye.com wrote:
On 11/26/18 10:58 AM, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote: > Here's the summary from the page, which proposes we pause the release > after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
Yeah, I think our goal should be 1 minute... just as realistic as one hour. ;)
I have not looked into things recently. However:
- The mock fsync change (https://pagure.io/releng/issue/7909) may help
some.
- We have plans to redo the setup on the s390x builders, which should
result in them being much faster. That should help.
- I know pungi maintaienrs have been doing some work to make things
faster (even in the last release out this morning).
All that said, I am not sure there's going to be a way to get it down to an hour. I think getting it down to 3-4hours may be very helpful tho and might be possible.
I totally agree, also as Peter mentioned, IOT composes are just taking 1 hour as its just consuming the repo rather than generating newRepo as the normal nightly composes does. If we can generate the newRepo on the fly whenever a package gets build and just use the repo to generate the artifacts either individually (splitting the compose) or parallely (as we are doing right now mostly) that would really help.
Unfortunately the newRepo/installer bit is likely the most I/O intensive, but I think it we split it out into it's own process, see mail I just sent, it would be a good start, and then also a good separate standalone focus for optimizations.
Peter
On Mon, Nov 26, 2018 at 1:59 PM Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
One of the large factors -- which I think was noted in the page -- was RPM scriptlets. Stephen Gallagher met with WIll Woods and James Antill just recently to talk about this. The time savings was substantial. Combining that with reducing the size of "speculative composes" that could be used for CI and other pre-ship purposes could get us to where we want to be. Stephen's carrying a heavy duty project at the moment but when that's done in a few weeks, this will be getting more attention.
The fact of the matter is that the compose has a LOT of I/O and I/O is slow and the cost to upgrade the infrastructure to get high through put I/O is expensive and no one has ever offered the budget to resolve that problem. The fact of the matter with this is that we now produce a lot of release artifacts and that takes time, some of it can probably be run more in parallel, and possibly some things run at different times but without a combination of investment in infrastructure, and maybe even hacking and slashing of components this is not simply a check box.
That's why the proposal is to skip an entire release to give us time to work on it. If it was easy, that wouldn't be necessary.
There are of course at least *two* possible solutions to "there's lots of I/O". One is "throw money at making the I/O faster", the other is "figure out how to do less I/O".
One hour is a *target*, not a *requirement*. I don't think anyone's going to consider the time wasted if we wind up getting it down to two hours instead of one.
(Though personally I'd like ten minutes. And two unicorns.)
Just so.
On Mon, Nov 26, 2018 at 7:52 PM Paul Frields stickster@gmail.com wrote:
On Mon, Nov 26, 2018 at 1:59 PM Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Nov 26, 2018 at 6:53 PM Adam Williamson adamwill@fedoraproject.org wrote:
On Mon, 2018-11-26 at 18:44 +0000, Peter Robinson wrote:
On Mon, Nov 26, 2018 at 4:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
Can I have a unicorn? Everyone wants this bug absolutely no one has done analysis.
I just don't believe this is true. I'm pretty sure Dennis and Kevin have both looked into it before.
Yes, I mean recently, I was involved in the last time we attempted this, and there was a number of things identified but quite a bit has changed since that last happened.
One of the large factors -- which I think was noted in the page -- was RPM scriptlets. Stephen Gallagher met with WIll Woods and James Antill just recently to talk about this. The time savings was substantial. Combining that with reducing the size of "speculative composes" that
There's no facts or details, to quote "The reported speed increase for transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson pbrobinson@gmail.com wrote:
There's no facts or details, to quote "The reported speed increase for transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
Paul was referring to image creation here. We should be able to get a fairly significant (at least one order of magnitude) speed-up on creating any image that runs an RPM installation transaction. I'm not sure how much that will save in the *whole* compose process, but as it's an identified spot where we know how to optimize it, we should do so.
On Mon, Nov 26, 2018 at 8:22 PM Stephen Gallagher sgallagh@redhat.com wrote:
On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson pbrobinson@gmail.com wrote:
There's no facts or details, to quote "The reported speed increase for transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
Paul was referring to image creation here. We should be able to get a fairly significant (at least one order of magnitude) speed-up on creating any image that runs an RPM installation transaction. I'm not sure how much that will save in the *whole* compose process, but as it's an identified spot where we know how to optimize it, we should do so.
But ultimately removing rpm transactions is an optimisation to rpm which, when it lands, the compose plus numerous other compoents that use that code path will just consume and at that point get the benefits of, it's not a reason to skip a release nor is it going to be a revolution that gets the composes down to an hour. I don't believe that that alone is and order of magnitude change, do you have any actual hard figured to back the statement up?
On Mon, Nov 26, 2018 at 3:34 PM Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Nov 26, 2018 at 8:22 PM Stephen Gallagher sgallagh@redhat.com wrote:
On Mon, Nov 26, 2018 at 3:03 PM Peter Robinson pbrobinson@gmail.com wrote:
There's no facts or details, to quote "The reported speed increase for transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
Paul was referring to image creation here. We should be able to get a fairly significant (at least one order of magnitude) speed-up on creating any image that runs an RPM installation transaction. I'm not sure how much that will save in the *whole* compose process, but as it's an identified spot where we know how to optimize it, we should do so.
But ultimately removing rpm transactions is an optimisation to rpm which, when it lands, the compose plus numerous other compoents that use that code path will just consume and at that point get the benefits of, it's not a reason to skip a release nor is it going to be a revolution that gets the composes down to an hour. I don't believe that that alone is and order of magnitude change, do you have any actual hard figured to back the statement up?
Paul overstated the general importance of the scriptlet work. As I said above, it will affect the RPM transactions. Will Woods has numbers on the actual performance gains that I'm sure he'll share on this thread at some point. I said above "I'm not sure how much that will save in the *whole* compose process" and I meant that. I'm not advocating that we postpone F31 solely for these changes. At the same time, let's be careful not to slip into a "perfect is the enemy of good" situation where we start critiquing individual solutions as not having enough of an effect.
On Mon, Nov 26, 2018 at 08:02:26PM +0000, Peter Robinson wrote:
transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
It seems like there's a lot of room to optimize those things too. For example, extract the needed metadata into a database and update that at build time rather than compose time, and run createrepo against that instead of rpms directly.
On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 26, 2018 at 08:02:26PM +0000, Peter Robinson wrote:
transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
It seems like there's a lot of room to optimize those things too. For example, extract the needed metadata into a database and update that at build time rather than compose time, and run createrepo against that instead of rpms directly.
Completely agree there's a LOT of improvement that can be done, but I don't see why that development work cannot be done in parallel and put it into production when each of the various different components are ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
On Mon, Nov 26, 2018 at 5:01 PM Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Mon, Nov 26, 2018 at 08:02:26PM +0000, Peter Robinson wrote:
transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
It seems like there's a lot of room to optimize those things too. For example, extract the needed metadata into a database and update that at build time rather than compose time, and run createrepo against that instead of rpms directly.
Completely agree there's a LOT of improvement that can be done, but I don't see why that development work cannot be done in parallel and put it into production when each of the various different components are
Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem".
ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area.
josh
El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió:
On Mon, Nov 26, 2018 at 5:01 PM Peter Robinson pbrobinson@gmail.com wrote:
On Mon, Nov 26, 2018 at 9:48 PM Matthew Miller < mattdm@fedoraproject.org> wrote:
On Mon, Nov 26, 2018 at 08:02:26PM +0000, Peter Robinson wrote:
transactions is substantial. This may solve (or greatly reduce) the compose speed problem, TBD." but it doesn't report the context of the "speed increase for transactions" .... like which transactions? There's a lot of I/O heavy processes in the compose, like createrepo, that don't run any rpm tracsactions what so ever.
It seems like there's a lot of room to optimize those things too. For example, extract the needed metadata into a database and update that at build time rather than compose time, and run createrepo against that instead of rpms directly.
Completely agree there's a LOT of improvement that can be done, but I don't see why that development work cannot be done in parallel and put it into production when each of the various different components are
Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem".
This is no longer entirely true, development of the compose tools is done entirely by PNT DevOps today. Bodhi and other tools in parts of the delivery pipeline are developed by people in Fedora Engineering who also have other tasks to do.
ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area.
We would need to engage with the team working on the compose tools and see what time they can dedicate to meeting Fedora's needs. Likely a end state needs to be defined. then the work can be scoped.
Dennis
On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore dennis@ausil.us wrote:
El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió:
Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem".
This is no longer entirely true, development of the compose tools is done entirely by PNT DevOps today. Bodhi and other tools in parts of the delivery pipeline are developed by people in Fedora Engineering who also have other tasks to do.
ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area.
We would need to engage with the team working on the compose tools and see what time they can dedicate to meeting Fedora's needs. Likely a end state needs to be defined. then the work can be scoped.
There are a lot of assumptions wrapped up in these statements. I'm not sure we should be assuming that delivery mechanisms ought to all be tied to internal Red Hat teams. This might be a good opportunity for us (meaning people variously assigned in both places) to think about whether we should try to decouple more of our delivery side (compose, push, etc.), while keeping a tight bond at the build/test side.
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
On Mon, Nov 26, 2018 at 7:45 PM Paul Frields stickster@gmail.com wrote:
On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore dennis@ausil.us wrote:
El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió:
Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem".
This is no longer entirely true, development of the compose tools is done entirely by PNT DevOps today. Bodhi and other tools in parts of the delivery pipeline are developed by people in Fedora Engineering who also have other tasks to do.
ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area.
We would need to engage with the team working on the compose tools and see what time they can dedicate to meeting Fedora's needs. Likely a end state needs to be defined. then the work can be scoped.
There are a lot of assumptions wrapped up in these statements. I'm not sure we should be assuming that delivery mechanisms ought to all be tied to internal Red Hat teams. This might be a good opportunity for us (meaning people variously assigned in both places) to think about whether we should try to decouple more of our delivery side (compose, push, etc.), while keeping a tight bond at the build/test side.
Sure. We still either need more people or to drop existing things to cover that with the existing people.
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
There are already a number of tools that are shared across Fedora and RHEL and I can tell you for a fact that many on the RHEL side would like to see the same or similar improvements that have been mentioned here. Let's start scoping the problems and solutions before we decide Fedora should go off and do it's own thing (again).
I'm interested in evolving more than just tooling. I continue to want to see Fedora and CentOS coming together more closely, so that we can leverage both of the amazing communities these projects provide. If we do that, we should be scoping how to do release tooling effectively across both Fedora and CentOS, while still keeping RHEL in mind as well. Having 3 (or more) different implementations of compose and release tooling seems ridiculous. I'd like to get it down to a community version and then have RHEL additions for product specific things if possible, driven by RHEL stakeholders. That way we get an actual upstream/downstream relationship with our tooling in addition to our distro.
josh
El lun, 26-11-2018 a las 19:44 -0500, Paul Frields escribió:
On Mon, Nov 26, 2018 at 5:47 PM Dennis Gilmore dennis@ausil.us wrote:
El lun, 26-11-2018 a las 17:14 -0500, Josh Boyer escribió:
Because the people that would be tasked with doing the development are also tasked with cranking out the release. It's a "need more people problem".
This is no longer entirely true, development of the compose tools is done entirely by PNT DevOps today. Bodhi and other tools in parts of the delivery pipeline are developed by people in Fedora Engineering who also have other tasks to do.
ready to go. Ultimately a lot of those sort of ways of optimising things are things that have been known for some time but no one has actually stepped up to do the dev work and it it into production shape. Stopping the standard distro work won't miraculously make that other work happen and appear.
Well, I kind of disagree. Barring magical people that appear out of the woodwork that know how the tools work and how they need to be improved, I don't see an alternative. Not doing a release to focus on our tech debt seems like a good tradeoff. If there are others that really WANT to continue cranking the release with the tools as they are today... that might be something that could be pursued. It would still require net new contributors to a critical area.
We would need to engage with the team working on the compose tools and see what time they can dedicate to meeting Fedora's needs. Likely a end state needs to be defined. then the work can be scoped.
There are a lot of assumptions wrapped up in these statements. I'm not sure we should be assuming that delivery mechanisms ought to all be tied to internal Red Hat teams. This might be a good opportunity for us (meaning people variously assigned in both places) to think about whether we should try to decouple more of our delivery side (compose, push, etc.), while keeping a tight bond at the build/test side.
There is a lot of assumptions in your assumptions, language is hard :), I was trying to outline where things are at not at all how things should be, sorry I was not clear enough there. My main goal was pointing out changes from where things were 5 years ago.
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
Compose tooling is largely shared, delivery tooling is not, though there is room for improvement and convergence on both sides. We should also consider CentOS and how we can share tooling, process and resources, they still use something else entirely. Everything used in Fedora is developed in the open and can be contributed to by anyone. We have been talking about making the whole process flexible and dynamic for years. I know internally they would like to have many of the same things Fedora wants.
Dennis
On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore dennis@ausil.us wrote: [...snip...]
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
Compose tooling is largely shared, delivery tooling is not, though there is room for improvement and convergence on both sides. We should also consider CentOS and how we can share tooling, process and resources, they still use something else entirely. Everything used in Fedora is developed in the open and can be contributed to by anyone. We have been talking about making the whole process flexible and dynamic for years. I know internally they would like to have many of the same things Fedora wants.
Yeah, it sounds like we're aiming in the same direction here. This is part of what I mean by "decomposing the compose." We have one rather monolithic process right now, all or nothing. And it does so much that it's impossible to get any fast results from it. That doesn't mean it doesn't work well for what it does. But it's not very flexible, as you pointed out.
With a more flexible compose we could choose to do smaller tasks (as well as capitalize on other time savings) for something like a speculative testing build. That kind of build could be used to perform integration testing as well as other tasks (automated, not manual!). We could even potentially test against much of the repetitive, time-consuming matrices manually (and heroically) run by QA. Then only choose to do more extensive tasks based on that being "green."
On Tue, Nov 27, 2018 at 6:26 PM Paul Frields stickster@gmail.com wrote:
On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore dennis@ausil.us wrote: [...snip...]
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
Compose tooling is largely shared, delivery tooling is not, though there is room for improvement and convergence on both sides. We should also consider CentOS and how we can share tooling, process and resources, they still use something else entirely. Everything used in Fedora is developed in the open and can be contributed to by anyone. We have been talking about making the whole process flexible and dynamic for years. I know internally they would like to have many of the same things Fedora wants.
Yeah, it sounds like we're aiming in the same direction here. This is part of what I mean by "decomposing the compose." We have one rather monolithic process right now, all or nothing. And it does so much that it's impossible to get any fast results from it. That doesn't mean it doesn't work well for what it does. But it's not very flexible, as you pointed out.
Actually amusingly if you look back through the history post F-21 and the massive delays that happened due to various non monolithic processes one of the outcomes of the "get a release done with Fedora.next" was to be able to do full composes every day because prior to that we only did newRepo, a network install and live/cloud images and often we'd get to Alpha TC1 right after branching to discover a whole bunch of stuff was completely broken. We finally got this with the move to "pungi 4" in Fedora 24, and in the intervening releases we had huge amounts of complaints because rel-eng couldn't provide all release artifacts to be pushed through CI/CD so regressions could be fixed instantaneously. Yes, I'm aware things change, and I often push them, but it's worth knowing the history how we got to where we are!
With a more flexible compose we could choose to do smaller tasks (as well as capitalize on other time savings) for something like a speculative testing build. That kind of build could be used to perform integration testing as well as other tasks (automated, not manual!).
Yes, and in some regards there's already parts of the project doing this, the TWA release runs like this, as does the IoT release and I mentioned above ways to break some of the bits up more.
We could even potentially test against much of the repetitive, time-consuming matrices manually (and heroically) run by QA. Then only choose to do more extensive tasks based on that being "green."
Completely agree, but there's quite a bit of work to be done to get to that unicorn but none of the process to get there has been documented or even proposed in this thread, it's not easy (believe me a lot of people have looked over the years I've been involved in rel-eng) and most of what I've seen of actual concrete means to get there won't get us anywhere close to that, while yes it will still improve it.
Peter
On Tue, 2018-11-27 at 19:30 +0000, Peter Robinson wrote:
On Tue, Nov 27, 2018 at 6:26 PM Paul Frields stickster@gmail.com wrote:
On Tue, Nov 27, 2018 at 9:50 AM Dennis Gilmore dennis@ausil.us wrote: [...snip...]
The customers RH serves have specific expectations, and in part that dictates how delivery tooling is done. Binding the community to that may be counterproductive. This is especially true now that RHEL 8 Beta is out -- it's the perfect time for us to be rethinking at that level.
Compose tooling is largely shared, delivery tooling is not, though there is room for improvement and convergence on both sides. We should also consider CentOS and how we can share tooling, process and resources, they still use something else entirely. Everything used in Fedora is developed in the open and can be contributed to by anyone. We have been talking about making the whole process flexible and dynamic for years. I know internally they would like to have many of the same things Fedora wants.
Yeah, it sounds like we're aiming in the same direction here. This is part of what I mean by "decomposing the compose." We have one rather monolithic process right now, all or nothing. And it does so much that it's impossible to get any fast results from it. That doesn't mean it doesn't work well for what it does. But it's not very flexible, as you pointed out.
Actually amusingly if you look back through the history post F-21 and the massive delays that happened due to various non monolithic processes one of the outcomes of the "get a release done with Fedora.next" was to be able to do full composes every day because prior to that we only did newRepo, a network install and live/cloud images and often we'd get to Alpha TC1 right after branching to discover a whole bunch of stuff was completely broken. We finally got this with the move to "pungi 4" in Fedora 24, and in the intervening releases we had huge amounts of complaints because rel-eng couldn't provide all release artifacts to be pushed through CI/CD so regressions could be fixed instantaneously. Yes, I'm aware things change, and I often push them, but it's worth knowing the history how we got to where we are!
Sure, I know about this (I did quite a bit of work around it). But I don't see any kind of inconsistency or anything here. Yes, we needed to ability to compose the whole distro in a saner and more organized way, which is what Pungi 4 did, which was a significant improvement. But it would *also* be extremely useful to be able to do smaller, faster subsets of the compose.
It's not like that's what we were doing before, after all. As you say, we still only did it *nightly*. And since it still involved a newRepo, it still took forever.
With a more flexible compose we could choose to do smaller tasks (as well as capitalize on other time savings) for something like a speculative testing build. That kind of build could be used to perform integration testing as well as other tasks (automated, not manual!).
Yes, and in some regards there's already parts of the project doing this, the TWA release runs like this, as does the IoT release and I mentioned above ways to break some of the bits up more.
I know about this too, and mentioned it in my initial mail in this subthread. But it's all been done in a manual, ad-hoc way: we just write entirely new Pungi config files for each new compose profile we come up with. There's no notion of modularity, of being able to actually define the individual elements and easily combine and recombine them. There's no notion of inputs mapped to outputs, either, really. I don't think this approach really scales much further than we're currently doing it; I don't think it'd be practical to (for instance) manually split the big 'Fedora' compose into 'Fedora-Live', 'Fedora-ISO', 'Fedora-ARM-Disk', etc. etc. etc. Trying to do that *just along the same lines as we have created the existing 'new' composes* would probably give us an unmaintainable mess.
On Mon, Nov 26, 2018 at 5:05 PM Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post.
Hi Matthew,
I initially basically ignored the previous E-Mail, because it didn't come from a person I knew, and - from quickly glancing over the contents - read like marketing fluff. I should have dug deeper there, but we all only have limited time.
Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
This is definitely getting more attention. Maybe the initial E-Mail should have had a less bland Subject line, too ;)
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- better tooling for non-base deliverables
- better metrics for everything
Those all sound like good things to work on.
- define a base platform -- Red Hat wants to focus resources here
This point though ... hmm. Also kind of mentioned on the linked wiki page:
#3 (...) Define a small, draft set of content for the Platform which can iterate over time
What exactly does that mean? Moving to something two-tiered like the ubuntu model (main / universe) - which I think is a terrible solution? (And what would be the names for this in fedora - "core" and "extras"? (pun definitely intended))
Also, I find it funny that - somehow - tons of resources would be available for "better CI pipeline tests for everything" (!), but we'd have to "focus resources" when it comes to the fedora "platform" itself.
I hope my response doesn't read overly negative or critical - but I think some fedora contributors (including me) have been weary of something like this ("focusing resources") since the IBM news got announced.
Fabio
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/Problem_statements and comment in this thread.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, 2018-11-26 at 20:03 +0100, Fabio Valentini wrote:
Also, I find it funny that - somehow - tons of resources would be available for "better CI pipeline tests for everything" (!), but we'd have to "focus resources" when it comes to the fedora "platform" itself.
These two sort of go together, I think: we don't truly expect to have 'better CI pipeline tests for EVERYTHING', as in *every package in the distro*, but rather one advantage of somehow defining the 'Platform' is that it gives us a *smaller* set of bits to focus the CI work on.
On Monday, November 26, 2018, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Nov 16, 2018 at 09:50:33AM -0500, Paul Frields wrote:
Here's the summary from the page, which proposes we pause the release after F30 for these efforts:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
Please read https://fedoraproject.org/wiki/Objectives/Lifecycle/ Problem_statements and comment in this thread.
I am not really convinced that delaying the release for that makes sense.
It's not like people will stop updating packages or that existing release tools will magically disappear (they have to work to keep existing versions running).
So people can work on those improvement and bring them in when ready but stopping everything for that is overkill.
Dne 26. 11. 18 v 17:03 Matthew Miller napsal(a):
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
My experience is that things are fixed when we are in real time stress. When we have more time, then people (including me) are trying to develop/invent new things.
I have a counter proposal: Make the release every week or month - this will force us to automate all release processes :)
Miroslav
On Tue, Nov 27, 2018 at 4:41 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 26. 11. 18 v 17:03 Matthew Miller napsal(a):
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
My experience is that things are fixed when we are in real time stress. When we have more time, then people (including me) are trying to develop/invent new things.
I have a counter proposal: Make the release every week or month - this will force us to automate all release processes :)
It's funny you mention that, because I could foresee one outcome of focusing on a core platform being that we do a more frequent release. If we can effectively machine-test more of that platform, we could do releases on a timescale of weeks instead of only twice a year. The Atomic WG does this already. This probably makes more sense for an ostree based release than constantly spinning large, multi-GB ISOs.
Matthew Miller wrote:
I know it was a big time-off holiday week in the US, but I expected a little more interest in this post. Perhaps it seemed like too much text to digest along with turkey and stuffing. :) I'm highlighting it with a subject reflecting the big, direct impact, and here's some other top-level proposals:
- embrace Taiga (an open source kanban tool) for project planning
- fix the compose speed (target: one hour!)
- really actually for real gated Rawhide
- better CI pipeline tests for everything
- define a base platform -- Red Hat wants to focus resources here
- better tooling for non-base deliverables
- better metrics for everything
And why do we need to stop the presses to do that? This is entirely orthogonal to distribution development and can be done in parallel.
Kevin Kofler
On Wed, 2018-11-28 at 18:19 +0100, Kevin Kofler wrote:
And why do we need to stop the presses to do that? This is entirely orthogonal to distribution development and can be done in parallel.
The problem is that the people who do the work to get the release done are the same people who would need to do the retooling work, so there is a resource conflict.
Update: We'll be having discussions over the next few weeks to figure out how we can accommodate Lifecycle work without interrupting the Fedora release cadence. That includes in Brno, where we can meet up f2f with people working on parallel projects in automation, CI, and so on. We may also be able to grab additional people to do work here -- so this doesn't automatically equate to "we have to do more with less/the same people." After talking with the Council as well as some folks who are depending on the cadence, like IoT, it was clear we need to look at this option. For now, I intend to remove the cadence change from the objective requirements, unless more specific reasons why it has to happen become clear.
On Wed, Nov 28, 2018 at 1:05 PM Randy Barlow bowlofeggs@fedoraproject.org wrote:
On Wed, 2018-11-28 at 18:19 +0100, Kevin Kofler wrote:
And why do we need to stop the presses to do that? This is entirely orthogonal to distribution development and can be done in parallel.
The problem is that the people who do the work to get the release done are the same people who would need to do the retooling work, so there is a resource conflict. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Sun, Jan 06, 2019 at 12:35:25PM -0500, Paul Frields wrote:
Update: We'll be having discussions over the next few weeks to figure out how we can accommodate Lifecycle work without interrupting the Fedora release cadence. That includes in Brno, where we can meet up f2f with people working on parallel projects in automation, CI, and so on. We may also be able to grab additional people to do work here -- so this doesn't automatically equate to "we have to do more with less/the same people." After talking with the Council as well as some folks who are depending on the cadence, like IoT, it was clear we need to look at this option. For now, I intend to remove the cadence change from the objective requirements, unless more specific reasons why it has to happen become clear.
It's getting to be time to lay out the F31 schedule. Let's go ahead with the assumption that we'll use the regular end-of-October target.
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
Because if we keep "no breaking updates in stable" policy, then Fedora won't be "first anymore". You can do this only if rawhide will be more popular between people.
On Tue, Nov 27, 2018, 03:34 Brendan Conoboy <blc@redhat.com wrote:
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
-- Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On 11/27/18 9:16 AM, Igor Gnatenko wrote:
Because if we keep "no breaking updates in stable" policy, then Fedora won't be "first anymore".
Users perceive your "first" as unstable and unreliable. There are plenty of examples of how e.g. FC30 was broken and still is.
One such example is you personal favorite: dnf.
You can do this only if rawhide will be more popular between people.
Rawhide will never be end-user usable. I personally consider rawhide end-user usability to be a hoax.
On Tue, Nov 27, 2018 at 9:39 AM Ralf Corsepius rc040203@freenet.de wrote:
On 11/27/18 9:16 AM, Igor Gnatenko wrote:
Because if we keep "no breaking updates in stable" policy, then Fedora won't be "first anymore".
Users perceive your "first" as unstable and unreliable. There are plenty of examples of how e.g. FC30 was broken and still is.
Then it means that we have to do releases, otherwise we will stay with one version with outdated software not just for 6 months but for one year.
One such example is you personal favorite: dnf.
You can do this only if rawhide will be more popular between people.
Rawhide will never be end-user usable. I personally consider rawhide end-user usability to be a hoax. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Mon, Nov 26, 2018 at 9:27 PM Brendan Conoboy blc@redhat.com wrote:
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
I'm going to have a longer post to reply about the overall topic brought up over Turkey Day, but for this specific point, I think that it would be a tremendous mistake if we allowed RHEL to influence us into slowing down the core of Fedora (see, I said the "bad" word!).
In the past few years, I've been around the block a bit, being involved in different distribution release philosophies and processes. And one thing that is common among all of them is that "stuff rolls downstream". That is, an action from the top part usually has cascading effects downward.
Most of us don't consider the "core"/"base platform" of Fedora to be the "top", but it is. All the decisions about how every package is built and shipped is affected by this. It would be incredibly dangerous for Fedora to take a RHEL-like approach to the core, because it means that we're deciding that the core is not a place for innovation and advancement.
Now, I'll admit I have a personal stake here: much of what *I* do these days is in that part of the stack, and having that frozen would kill my ability to participate in the development of the distribution. But I'd like us to become less conservative instead of more so. I've been playing with the RHEL 8 beta for a bit now, and I have a pretty good idea what constitutes a "base" system in RHEL, and I'm afraid that it would be terrible if we froze that for a year.
However, to the larger point about lifecycles and supporting hardware makers with a Fedora LTS that works for 4 years, this could be a good focus point for a new "Fedora Long-Term" WG that would work to take selected releases of Fedora and stabilize them for a period of time. This would be similar to the original Fedora Legacy and openSUSE Evergreen[1] projects. An interesting twist here is that we could take some inspiration from Debian on how they run their LTS and allow direct sponsorship for releases to be supported past their original life[2]. That means that under the auspices of the Fedora Project, interested volunteers and commercial entities could come together and maintain a release with security fixes past the 13 month life, for an additional 2 years.
This approach has some advantages: notably only releases people are interested it would qualify, and the work burden scales with interest and sponsorship. The main disadvantage is that it won't happen "for free", which I think is actually a good thing here.
Even Red Hat could participate in this program, though I fully expect that they won't, since they have their hands full with Red Hat Enterprise Linux. But it'd be nice if they did, because it could also be an avenue to improve the transparency between Fedora and RHEL, and give a more direct relationship (and potential for influencing improvements to RHEL that we don't have today!). It would also make it easier for the 22 other product projects that Red Hat has to use Fedora as their upstream rather than CentOS, since some folks in those communities complain Fedora is too fast for them (*hack*RDO*cough*).
[1]: https://en.opensuse.org/openSUSE:Evergreen [2]: https://wiki.debian.org/LTS/
On Mon, Nov 26, 2018 at 8:26 PM Brendan Conoboy blc@redhat.com wrote:
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
Long cycles have been done before, and will be done again, it has been 4 or 5 years since the last one. I think skipping to a yearly cadence for every release isn't such a great idea. There are benefits to the cadence we have, but I do think perhaps a plan to regularly do this might be a good thing. Just as a ballpark, perhaps once every 5 releases. This would allow people to actually plan the big workloads for this timeframe. F30 is a 1 year, the next is F35. Once F31 is done, I can start planning the big project for F36. Of course it could also be argued that we only do a long release when needed, but this doesn't really allow us to plan as well, and doesn't set expectations for the community in the same way.
Justin
On Tue, Nov 27, 2018 at 8:21 AM Justin Forbes jmforbes@linuxtx.org wrote:
Long cycles have been done before, and will be done again, it has been 4 or 5 years since the last one. I think skipping to a yearly cadence for every release isn't such a great idea. There are benefits to the cadence we have, but I do think perhaps a plan to regularly do this might be a good thing.
Agreed. Doing a long cycle is a tradeoff. In general, the six month cycle works for what we're trying to do. But having the occasional long cycle gives us some space to work on bigger projects or things that can't be done while we're also trying to get a release out the door. With unlimited funding and people, we wouldn't have to make this tradeoff, but we do have limits.
I'm sympathetic to the argument that we should do it as needed, but my preference is to schedule it. This allows our community to plan ahead, both for the work to be done during the long cycle and also for things that should get done before and after. Predictability is helpful.
If we do say, for example, every 8th release will be a long cycle there's nothing that would prevent us from doing an unscheduled long cycle if we have to. But it would have to be a very compelling reason knowing that we'd have another long release coming up.
On 11/27/18 7:54 AM, Ben Cotton wrote:
On Tue, Nov 27, 2018 at 8:21 AM Justin Forbes jmforbes@linuxtx.org wrote:
Long cycles have been done before, and will be done again, it has been 4 or 5 years since the last one. I think skipping to a yearly cadence for every release isn't such a great idea. There are benefits to the cadence we have, but I do think perhaps a plan to regularly do this might be a good thing.
Agreed. Doing a long cycle is a tradeoff. In general, the six month cycle works for what we're trying to do. But having the occasional long cycle gives us some space to work on bigger projects or things that can't be done while we're also trying to get a release out the door. With unlimited funding and people, we wouldn't have to make this tradeoff, but we do have limits.
I'm sympathetic to the argument that we should do it as needed, but my preference is to schedule it. This allows our community to plan ahead, both for the work to be done during the long cycle and also for things that should get done before and after. Predictability is helpful.
If we do say, for example, every 8th release will be a long cycle there's nothing that would prevent us from doing an unscheduled long cycle if we have to. But it would have to be a very compelling reason knowing that we'd have another long release coming up.
The other way to slice it is having some parts of the distribution be on a longer cycle and others on a shorter cycle. If there is a 1 year gap between F30 and F31, perhaps the rules for what can be updated in F30 could be revised to maintain the value people get from the 6 month release cycle.
On Mon, Nov 26, 2018 at 9:26 PM Brendan Conoboy blc@redhat.com wrote:>
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
If we go ahead and skip the F31 release, I see this as a (perhaps necessary) desperation move - there is a small group of a dozen or so people who would be key to improving our release processes, but those people are always busy making releases.
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
- Owen
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
On Mon, Nov 26, 2018 at 9:26 PM Brendan Conoboy blc@redhat.com wrote:>
On 11/16/18 7:50 AM, Paul Frields wrote: [snip]
We should skip the F31 release cycle and leave F30 in place longer in order to focus on improving the tooling and testing changes. These tooling changes will improve the overall reliability of Fedora, and will decrease the manual effort and complexities involved in producing the distribution artifacts. Although we’ve done this before to make “editions” happen, the intent is to track this multi-team effort more actively so we can (1) use the time as well as possible, and (2) give the work maximum transparency.
If there is going to be a pause F30 seems like a good place to do it: New glibc, new compiler- and a full year for them to mature. It's a nice basis for a stable platform. What would the update policy be for this year- same as today? It seems like you're proposing this as a one-time event to pay down technical debt, which is great, but would you perhaps consider doing the same thing for F31, F32, etc? The basic reasons for technical debt will continue- why not plan to service the debt regularly?
If we go ahead and skip the F31 release, I see this as a (perhaps necessary) desperation move - there is a small group of a dozen or so people who would be key to improving our release processes, but those people are always busy making releases.
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
josh
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
Owen
On Tue, 27 Nov 2018 at 17:23, Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
I think this is another important problem we have, it is very difficult for someone interested to help in these domains. So we end up with a dozen of persons being heroic because they are actually the one that have the correct permissions and knowledge to do most of the work. We should definitely look at solving this problem.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
Owen _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Nov 27, 2018 at 11:21 AM Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
I would say it will be a learning curve and there will be complaints because it's change and people seem to only want change if it's the change they want or otherwise doesn't impact them. That's going to be a big deal to some :) Generally though, yes. We need to distribute the burden to the package sets and whomever maintains them.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
Ah, good!
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
That would be a concern to me as well, assuming we stuck with a *single* cadence and a *single* platform. With the lifecycle Objective, aren't we targeting multiple cadences and lifecycles? I mean, we have this today with Fedora Atomic/CoreOS vs. "regular" Fedora. Maybe Brendan was thinking singular, but I would very much like the ability in our tooling a process to have both the faster moving cadence of today AND a slower moving one. That's another reason I view a temporary halt as worthwhile. If done correctly, it enables Fedora to fill more usecases and lifecycles than we could ever accomplish with a single release.
josh
On 11/27/18 10:13 AM, Josh Boyer wrote:
On Tue, Nov 27, 2018 at 11:21 AM Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
I would say it will be a learning curve and there will be complaints because it's change and people seem to only want change if it's the change they want or otherwise doesn't impact them. That's going to be a big deal to some :) Generally though, yes. We need to distribute the burden to the package sets and whomever maintains them.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
Ah, good!
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
That would be a concern to me as well, assuming we stuck with a *single* cadence and a *single* platform. With the lifecycle Objective, aren't we targeting multiple cadences and lifecycles? I mean, we have this today with Fedora Atomic/CoreOS vs. "regular" Fedora. Maybe Brendan was thinking singular, but I would very much like the ability in our tooling a process to have both the faster moving cadence of today AND a slower moving one. That's another reason I view a temporary halt as worthwhile. If done correctly, it enables Fedora to fill more usecases and lifecycles than we could ever accomplish with a single release.
Josh has it: I am indeed thinking about how this works with the lifecycle objective. When we think about things as all or nothing we rapidly get into a zero sum game: Let's make this part of Fedora worse to make this other part better. We don't have to do that, we can just make Fedora better. That's what having multiple cadences and lifecycles is all about. So it is important to talk about how the rules would change if F30 takes a year, because if the rules don't change it might diminish one of the great things about Fedora.
We can definitely talk about whether moving to a slower cadence for certain parts of the base platform. But people don't judge Fedora on how beautifully we maintain glibc and gcc - they mostly judge it by installing it on a laptop and seeing how well it works. And it doesn't really matter how reliably we can create minimal container images if the workstation image doesn't compose or doesn't log in.
We need clear naming for the workstation releases we do. If we want to call them F30, F30.1, F31, F31.1 we can, but that doesn't inherently reduce load on releng or remove the need for infrastructure freezes ... my assumption was that the idea of delaying F31 is to actually *not* do an operating system release for a year, and that's something that I don't think we should do on an ongoing basis.
Owen
On Tue, Nov 27, 2018 at 1:19 PM Brendan Conoboy blc@redhat.com wrote:
On 11/27/18 10:13 AM, Josh Boyer wrote:
On Tue, Nov 27, 2018 at 11:21 AM Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
I would say it will be a learning curve and there will be complaints because it's change and people seem to only want change if it's the change they want or otherwise doesn't impact them. That's going to be a big deal to some :) Generally though, yes. We need to distribute the burden to the package sets and whomever maintains them.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
Ah, good!
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
That would be a concern to me as well, assuming we stuck with a *single* cadence and a *single* platform. With the lifecycle Objective, aren't we targeting multiple cadences and lifecycles? I mean, we have this today with Fedora Atomic/CoreOS vs. "regular" Fedora. Maybe Brendan was thinking singular, but I would very much like the ability in our tooling a process to have both the faster moving cadence of today AND a slower moving one. That's another reason I view a temporary halt as worthwhile. If done correctly, it enables Fedora to fill more usecases and lifecycles than we could ever accomplish with a single release.
Josh has it: I am indeed thinking about how this works with the lifecycle objective. When we think about things as all or nothing we rapidly get into a zero sum game: Let's make this part of Fedora worse to make this other part better. We don't have to do that, we can just make Fedora better. That's what having multiple cadences and lifecycles is all about. So it is important to talk about how the rules would change if F30 takes a year, because if the rules don't change it might diminish one of the great things about Fedora.
-- Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Tue, Nov 27, 2018 at 1:40 PM Owen Taylor otaylor@redhat.com wrote:
We can definitely talk about whether moving to a slower cadence for certain parts of the base platform. But people don't judge Fedora on how beautifully we maintain glibc and gcc - they mostly judge it by installing it on a laptop and seeing how well it works. And it doesn't really matter how reliably we can create minimal container images if the workstation image doesn't compose or doesn't log in.
We need clear naming for the workstation releases we do. If we want to call them F30, F30.1, F31, F31.1 we can, but that doesn't inherently reduce load on releng or remove the need for infrastructure freezes ... my assumption was that the idea of delaying F31 is to actually *not* do an operating system release for a year, and that's something that I don't think we should do on an ongoing basis.
Heard, understood, and agreed. Shortening the compose is a root problem to fix if we want to be able to judge reliability of what we want to release. That way we can test it more or less constantly.
As for freezes... right now we freeze for a total of roughly 2-3 months out of 12. We can avoid that with a pause -- at least in part, and maybe in total if we finish enough work to minimize freeze coming out of it. That's why taking the break makes sense.
Reclaiming that time, year after year, should be a knock-on effect of being able to release more frequently as a non-event. Then there's less (maybe no) reason to freeze -- the worst thing that happens is we pull the lever the next day. So a healthy part of our investment should be in minimizing recovery time.
On 11/27/2018 10:39 AM, Owen Taylor wrote:
We can definitely talk about whether moving to a slower cadence for certain parts of the base platform. But people don't judge Fedora on how beautifully we maintain glibc and gcc - they mostly judge it by installing it on a laptop and seeing how well it works. And it doesn't really matter how reliably we can create minimal container images if the workstation image doesn't compose or doesn't log in.
I'm not sure that that's fair. I mean, I judge Fedora on how much its features are likely to cause problems 9-36 months down the road in an EL release... or now, in the case of cross-distro integration and support. I could care less about GNOME, but care very much about gcc, glibc, the kernel, and packaging of libraries.
*Laptop users* certainly judge Fedora on how well it works on laptops, and Workstation environment users judge Fedora on the Workstation spin.
Any other statement would pre-suppose the exact answers this and related discussion would seem to be working towards.
-jc
On 11/27/18 11:39 AM, Owen Taylor wrote:
We can definitely talk about whether moving to a slower cadence for certain parts of the base platform. But people don't judge Fedora on how beautifully we maintain glibc and gcc - they mostly judge it by installing it on a laptop and seeing how well it works. And it doesn't really matter how reliably we can create minimal container images if the workstation image doesn't compose or doesn't log in.
It's true that the #1 use case for Fedora is the desktop, but that isn't all Fedora is used for. If Fedora is going to continue to grow it needs to serve more uses. At the same time it needs to keep doing a great job as a desktop- so we have to look at solutions that promote both.
How beautifully we maintain platform pieces like gcc, glibc, and other commonly used shared libraries is actually tremendously important to many users, to many developers, because it affects their ability to adopt Fedora as a base to do the things they want to do.
We need clear naming for the workstation releases we do. If we want to call them F30, F30.1, F31, F31.1 we can, but that doesn't inherently reduce load on releng or remove the need for infrastructure freezes ... my assumption was that the idea of delaying F31 is to actually *not* do an operating system release for a year, and that's something that I don't think we should do on an ongoing basis.
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
Owen
On Tue, Nov 27, 2018 at 1:19 PM Brendan Conoboy blc@redhat.com wrote:
On 11/27/18 10:13 AM, Josh Boyer wrote:
On Tue, Nov 27, 2018 at 11:21 AM Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 10:13 AM Owen Taylor otaylor@redhat.com wrote:
But for the next thousand or so Fedora developers, the release cycle is actually not a big deal - not something that takes much of their time - and it gives them a regular place to land feature work. And Fedora users appreciate a timely updated operating system (without having random rebases trickle in.)
It's not a big deal because they don't participate in making the release nor do they really know about all the duct tape and bailing wire holding all the machinery in place. Just like most of them don't appreciate the heroics involved from Fedora QA and rel-eng and the dozen or so people that regularly go well beyond the extra mile to get it out the door. This isn't done out of malice on their part. It's mostly that they just don't see it.
If we distribute this work out across the 1000 developers, we'll make life better for heroes, and it's *still* not going to be a big deal for the 1000 developers.
I would say it will be a learning curve and there will be complaints because it's change and people seem to only want change if it's the change they want or otherwise doesn't impact them. That's going to be a big deal to some :) Generally though, yes. We need to distribute the burden to the package sets and whomever maintains them.
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
Ah, good!
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
That would be a concern to me as well, assuming we stuck with a *single* cadence and a *single* platform. With the lifecycle Objective, aren't we targeting multiple cadences and lifecycles? I mean, we have this today with Fedora Atomic/CoreOS vs. "regular" Fedora. Maybe Brendan was thinking singular, but I would very much like the ability in our tooling a process to have both the faster moving cadence of today AND a slower moving one. That's another reason I view a temporary halt as worthwhile. If done correctly, it enables Fedora to fill more usecases and lifecycles than we could ever accomplish with a single release.
Josh has it: I am indeed thinking about how this works with the lifecycle objective. When we think about things as all or nothing we rapidly get into a zero sum game: Let's make this part of Fedora worse to make this other part better. We don't have to do that, we can just make Fedora better. That's what having multiple cadences and lifecycles is all about. So it is important to talk about how the rules would change if F30 takes a year, because if the rules don't change it might diminish one of the great things about Fedora.
-- Brendan Conoboy / RHEL Development Coordinator / Red Hat, Inc. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
regards,
bex
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together.
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 12:12 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I've failed to be clear - sorry about that. Let me try again.
Please remember that I tend think from the lens of user space, not kernel space. So there may be detail errors in this, I am hoping the concepts are valid though.
In general, I can run various versions of my applications against multiple different kernels, for example. Therefore, if I have a kernel that changed once a year, it isn't going to, for many applications, stop me from changing versions multiple times during the year. Therefore if Fedora had a stabilized bootable base, I could move my applications at the cadence of upstream, or a stabilized release (not at all) or at the speed in the middle I want. Fedora might not build the entire range of that, but I am not prevented from choosing amongst multiple Fedora provided options (stable vs devel or all supported upstream releases, for example).
The bootable base would change based on Fedora's needs. Perhaps we decide want to new kernels (again sorry for my failings in this field) every 6 months to introduce new drivers and hardware support. Someone who wants faster can self-build for their community/needs more frequently or a hardware vendor might want a kernel that doesn't change as often and is backported. Fedora may not build the whole range here either.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
"Eliminate the compilers" was meant to mean, make them modules or non-bootable base components as much as possible. I realize that for something like glibc that can be hard to impossible and for things like Fortran a non-issue.
Communities have different needs, and every time we freeze or force change we create challenges for those needs to be met. My point is that by keeping our base to the "light up the machine level (another hated idiom) and get containers/flatpaks/etc running" we can allow the builder to focus on their community's needs or the user to get their own too-fast/too-slow balance.
What I'd like to forego is the long "what is base" conversation. I'd pull it back to what does it take to boot the machine and get containers/flatpaks/modules running. Everything after that should be flexible, even if we put down requirements and rules for the use cases we want to tackle and the pieces we build and deliver.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Marketing Koolaid (in the interesting and love sense, not the pejorative) would be really good for Fedora right now, if we want increased adoption and contribution. Flexibility to allow our mission statement to keep being true is critical. I fully admit that my skills are not in distribution building and that you and others in this thread have those skills in greater capacity than I do. But I don't hear anyone starting from the premise of our mission statement and then moving forward. I feel like a build/distribution focused on easing the build process and ability to provide across the full too-fast/too-slow spectrum, even if we as a project don't fill the whole spectrum, is crucial to Fedora's ongoing success.
regards,
bex
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 1:14 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:12 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I've failed to be clear - sorry about that. Let me try again.
Please remember that I tend think from the lens of user space, not kernel space. So there may be detail errors in this, I am hoping the concepts are valid though.
In general, I can run various versions of my applications against multiple different kernels, for example. Therefore, if I have a kernel that changed once a year, it isn't going to, for many applications, stop me from changing versions multiple times during the year. Therefore if Fedora had a stabilized bootable base, I could move my applications at the cadence of upstream, or a stabilized release (not at all) or at the speed in the middle I want. Fedora might not build the entire range of that, but I am not prevented from choosing amongst multiple Fedora provided options (stable vs devel or all supported upstream releases, for example).
So you mean that you want CentOS stable base and latest software you want?
The bootable base would change based on Fedora's needs. Perhaps we decide want to new kernels (again sorry for my failings in this field) every 6 months to introduce new drivers and hardware support. Someone who wants faster can self-build for their community/needs more frequently or a hardware vendor might want a kernel that doesn't change as often and is backported. Fedora may not build the whole range here either.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
"Eliminate the compilers" was meant to mean, make them modules or non-bootable base components as much as possible. I realize that for something like glibc that can be hard to impossible and for things like Fortran a non-issue.
Communities have different needs, and every time we freeze or force change we create challenges for those needs to be met. My point is that by keeping our base to the "light up the machine level (another hated idiom) and get containers/flatpaks/etc running" we can allow the builder to focus on their community's needs or the user to get their own too-fast/too-slow balance.
What I'd like to forego is the long "what is base" conversation. I'd pull it back to what does it take to boot the machine and get containers/flatpaks/modules running. Everything after that should be flexible, even if we put down requirements and rules for the use cases we want to tackle and the pieces we build and deliver.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Marketing Koolaid (in the interesting and love sense, not the pejorative) would be really good for Fedora right now, if we want increased adoption and contribution. Flexibility to allow our mission statement to keep being true is critical. I fully admit that my skills are not in distribution building and that you and others in this thread have those skills in greater capacity than I do. But I don't hear anyone starting from the premise of our mission statement and then moving forward. I feel like a build/distribution focused on easing the build process and ability to provide across the full too-fast/too-slow spectrum, even if we as a project don't fill the whole spectrum, is crucial to Fedora's ongoing success.
I have nothing against, but stopping release for some mysterious reasons seems bad idea, but if we get some specific goals which we try to achieve -- I might even support this idea.
regards,
bex
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-- Brian (bex) Exelbierd | bexelbie@redhat.com | bex@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 1:31 PM Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
On Wed, Nov 28, 2018 at 1:14 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:12 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I've failed to be clear - sorry about that. Let me try again.
Please remember that I tend think from the lens of user space, not kernel space. So there may be detail errors in this, I am hoping the concepts are valid though.
In general, I can run various versions of my applications against multiple different kernels, for example. Therefore, if I have a kernel that changed once a year, it isn't going to, for many applications, stop me from changing versions multiple times during the year. Therefore if Fedora had a stabilized bootable base, I could move my applications at the cadence of upstream, or a stabilized release (not at all) or at the speed in the middle I want. Fedora might not build the entire range of that, but I am not prevented from choosing amongst multiple Fedora provided options (stable vs devel or all supported upstream releases, for example).
So you mean that you want CentOS stable base and latest software you want?
Nope. I want CentOS to serve their user base. If the CentOS project chooses to formally become part of Fedora and to release a longer-maintained base to their users under the banner of our mission, I welcome them with open arms. But, that is their decision.
I don't think Fedora wants to get into 10 year life cycles and that level of backporting. But, accepting a new kernel/bootable base shouldn't force me to take a new version of LibreOffice, for example.
The bootable base would change based on Fedora's needs. Perhaps we decide want to new kernels (again sorry for my failings in this field) every 6 months to introduce new drivers and hardware support. Someone who wants faster can self-build for their community/needs more frequently or a hardware vendor might want a kernel that doesn't change as often and is backported. Fedora may not build the whole range here either.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
"Eliminate the compilers" was meant to mean, make them modules or non-bootable base components as much as possible. I realize that for something like glibc that can be hard to impossible and for things like Fortran a non-issue.
Communities have different needs, and every time we freeze or force change we create challenges for those needs to be met. My point is that by keeping our base to the "light up the machine level (another hated idiom) and get containers/flatpaks/etc running" we can allow the builder to focus on their community's needs or the user to get their own too-fast/too-slow balance.
What I'd like to forego is the long "what is base" conversation. I'd pull it back to what does it take to boot the machine and get containers/flatpaks/modules running. Everything after that should be flexible, even if we put down requirements and rules for the use cases we want to tackle and the pieces we build and deliver.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Marketing Koolaid (in the interesting and love sense, not the pejorative) would be really good for Fedora right now, if we want increased adoption and contribution. Flexibility to allow our mission statement to keep being true is critical. I fully admit that my skills are not in distribution building and that you and others in this thread have those skills in greater capacity than I do. But I don't hear anyone starting from the premise of our mission statement and then moving forward. I feel like a build/distribution focused on easing the build process and ability to provide across the full too-fast/too-slow spectrum, even if we as a project don't fill the whole spectrum, is crucial to Fedora's ongoing success.
I have nothing against, but stopping release for some mysterious reasons seems bad idea, but if we get some specific goals which we try to achieve -- I might even support this idea.
I am not sure releases as a construct, are as useful in their current form to us today as they were in the past. I (emphatically) do NOT want us to stop producing and releasing new software. In fact, I want it released faster and more often, but without the accompanying user disruption of forced upgrade/rolling releases. I want our editions/spins/labs/remixes/etc. to be able to assemble the Fedora that meet's their communities needs from the amazing pool of contributions the project as a whole has.
Fedora may not build or even endorse everything that is made, but this allows our mission statement to be put into practice. It allows Fedora to become the easiest distribution to base your solution on.
regards,
bex
regards,
bex
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-- Brian (bex) Exelbierd | bexelbie@redhat.com | bex@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 1:49 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 1:31 PM Igor Gnatenko ignatenkobrain@fedoraproject.org wrote:
On Wed, Nov 28, 2018 at 1:14 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:12 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I've failed to be clear - sorry about that. Let me try again.
Please remember that I tend think from the lens of user space, not kernel space. So there may be detail errors in this, I am hoping the concepts are valid though.
In general, I can run various versions of my applications against multiple different kernels, for example. Therefore, if I have a kernel that changed once a year, it isn't going to, for many applications, stop me from changing versions multiple times during the year. Therefore if Fedora had a stabilized bootable base, I could move my applications at the cadence of upstream, or a stabilized release (not at all) or at the speed in the middle I want. Fedora might not build the entire range of that, but I am not prevented from choosing amongst multiple Fedora provided options (stable vs devel or all supported upstream releases, for example).
So you mean that you want CentOS stable base and latest software you want?
Nope. I want CentOS to serve their user base. If the CentOS project chooses to formally become part of Fedora and to release a longer-maintained base to their users under the banner of our mission, I welcome them with open arms. But, that is their decision.
I don't think Fedora wants to get into 10 year life cycles and that level of backporting. But, accepting a new kernel/bootable base shouldn't force me to take a new version of LibreOffice, for example.
So what you are saying here is what "first" version of Modularity was about. And it didn't work out because of many reasons. One of them is no useful CI infrastructure (still not solved) which didn't allow packagers to remove useless BuildRequires for tests (because rpmbuild is still tte way to run test suite).
If we can get commitment from people who maintain "minimal viable base" to move their tests to CI and get FTBFS fixes in time -- it might work.
The bootable base would change based on Fedora's needs. Perhaps we decide want to new kernels (again sorry for my failings in this field) every 6 months to introduce new drivers and hardware support. Someone who wants faster can self-build for their community/needs more frequently or a hardware vendor might want a kernel that doesn't change as often and is backported. Fedora may not build the whole range here either.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
"Eliminate the compilers" was meant to mean, make them modules or non-bootable base components as much as possible. I realize that for something like glibc that can be hard to impossible and for things like Fortran a non-issue.
Communities have different needs, and every time we freeze or force change we create challenges for those needs to be met. My point is that by keeping our base to the "light up the machine level (another hated idiom) and get containers/flatpaks/etc running" we can allow the builder to focus on their community's needs or the user to get their own too-fast/too-slow balance.
What I'd like to forego is the long "what is base" conversation. I'd pull it back to what does it take to boot the machine and get containers/flatpaks/modules running. Everything after that should be flexible, even if we put down requirements and rules for the use cases we want to tackle and the pieces we build and deliver.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Marketing Koolaid (in the interesting and love sense, not the pejorative) would be really good for Fedora right now, if we want increased adoption and contribution. Flexibility to allow our mission statement to keep being true is critical. I fully admit that my skills are not in distribution building and that you and others in this thread have those skills in greater capacity than I do. But I don't hear anyone starting from the premise of our mission statement and then moving forward. I feel like a build/distribution focused on easing the build process and ability to provide across the full too-fast/too-slow spectrum, even if we as a project don't fill the whole spectrum, is crucial to Fedora's ongoing success.
I have nothing against, but stopping release for some mysterious reasons seems bad idea, but if we get some specific goals which we try to achieve -- I might even support this idea.
I am not sure releases as a construct, are as useful in their current form to us today as they were in the past. I (emphatically) do NOT want us to stop producing and releasing new software. In fact, I want it released faster and more often, but without the accompanying user disruption of forced upgrade/rolling releases. I want our editions/spins/labs/remixes/etc. to be able to assemble the Fedora that meet's their communities needs from the amazing pool of contributions the project as a whole has.
Fedora may not build or even endorse everything that is made, but this allows our mission statement to be put into practice. It allows Fedora to become the easiest distribution to base your solution on.
Yes, I fully support this.
regards,
bex
regards,
bex
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-- Brian (bex) Exelbierd | bexelbie@redhat.com | bex@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-- Brian (bex) Exelbierd | bexelbie@redhat.com | bex@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 1:07 PM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:12 PM Peter Robinson pbrobinson@gmail.com wrote:
On Wed, Nov 28, 2018 at 8:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
On Wed, Nov 28, 2018 at 4:15 AM Brendan Conoboy blc@redhat.com wrote:
Paul's proposal was definitely a one-time pause for the reasons you state. He requested we follow-up with additional questions and suggestions so I'm questioning and suggesting taking it a step further. We talk about rolling releases, but people are skeptical due to rawhide instability. What does it look like if the "rolling" happens on top of an otherwise stable platform where fundamentals like compilers, libraries and core system tools are held steady, but things on top move fast? Maybe you don't need an F30.1, maybe it means F30 just keeps getting nice incremental updates for as long as the editions want to stick with it. Variable lifecycle or cadence can open up these kinds of options. Some things are better fast. Some things are better slow.
This. Yes This. +100
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The above doesn't make sense, you're saying "move as fast as upstream" and "a base that doesn't change" in the same context, which is it?
I've failed to be clear - sorry about that. Let me try again.
Please remember that I tend think from the lens of user space, not kernel space. So there may be detail errors in this, I am hoping the concepts are valid though.
In general, I can run various versions of my applications against multiple different kernels, for example. Therefore, if I have a kernel that changed once a year, it isn't going to, for many applications, stop me from changing versions multiple times during the year. Therefore if Fedora had a stabilized bootable base, I could move my applications at the cadence of upstream, or a stabilized release (not at all) or at the speed in the middle I want. Fedora might not build the entire range of that, but I am not prevented from choosing amongst multiple Fedora provided options (stable vs devel or all supported upstream releases, for example).
I think the kernel is a bad example here. It's exceedly stable across releases, so it's probably the one component that's least problematic to upgrade during a fedora release. The fedora kernel team is already doing that, and they are doing an excellent job, which they definitely deserve more praise for than they get. For me, the frequent, well-executed stable kernel updates are one thing that positively distinguishes fedora from other non-rolling distros.
On the other hand, specific releases of GCC, glibc, (golang, ruby, python, ocaml, perl, node, insert other compilers / interpreters here) would be better examples of a "stable base". They really warrant "new releases", since major updates of those components usually require mass rebuilds of their dependent packages.
So, I'd rather draw the line between "base" and "rolling application releases" *atop* the development environment that the versions of GCC, glibc, python, golang, etc. make up, not below them. This is what many packagers already do, from my experience. GNOME really is an exception here, even KDE/KF5/Plasma is updated regularly during stable releases.
Whether the "base" system (that's used for development and building all other packages atop it) is released every 6 months (aligning with semi-annual releases of i.e. golang) or yearly (more in line with ~annual releases of GCC, python, etc.) isn't as important then, since what's important to users (hardware support - kernel, and applications) is updated independent from the development base. (As I said, that's what many packagers - including myself - already do within the current system.)
Fabio
The bootable base would change based on Fedora's needs. Perhaps we decide want to new kernels (again sorry for my failings in this field) every 6 months to introduce new drivers and hardware support. Someone who wants faster can self-build for their community/needs more frequently or a hardware vendor might want a kernel that doesn't change as often and is backported. Fedora may not build the whole range here either.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Again what do you even mean by eliminate the compilers? Also how do we not change something core, such as a compiler, for 4 years while also change it every 30 days?
"Eliminate the compilers" was meant to mean, make them modules or non-bootable base components as much as possible. I realize that for something like glibc that can be hard to impossible and for things like Fortran a non-issue.
Communities have different needs, and every time we freeze or force change we create challenges for those needs to be met. My point is that by keeping our base to the "light up the machine level (another hated idiom) and get containers/flatpaks/etc running" we can allow the builder to focus on their community's needs or the user to get their own too-fast/too-slow balance.
What I'd like to forego is the long "what is base" conversation. I'd pull it back to what does it take to boot the machine and get containers/flatpaks/modules running. Everything after that should be flexible, even if we put down requirements and rules for the use cases we want to tackle and the pieces we build and deliver.
I recognize that I, like most people, see the world through the lens of my specific use case, but remember, "Fedora creates an innovative platform for hardware, clouds, and containers that enables software developers and community members to build tailored solutions for their users." As long as we don't block a use cases arbitrarily and we leave room for that innovation and service we are doing the right thing. The debate about what use cases should be done fully by Fedora, enabled for a SIG/WG via our build system or done externally by those using only the parts that make sense for them is a separate debate.
I agree on some of your points above, but TBH most of it reads like some form of marketing coolaid!
Marketing Koolaid (in the interesting and love sense, not the pejorative) would be really good for Fedora right now, if we want increased adoption and contribution. Flexibility to allow our mission statement to keep being true is critical. I fully admit that my skills are not in distribution building and that you and others in this thread have those skills in greater capacity than I do. But I don't hear anyone starting from the premise of our mission statement and then moving forward. I feel like a build/distribution focused on easing the build process and ability to provide across the full too-fast/too-slow spectrum, even if we as a project don't fill the whole spectrum, is crucial to Fedora's ongoing success.
regards,
bex
Also it does account at all for any and/or all of the resources that we'd need to even enact some of this, even if it's possible?
0: Builder's desires are the desires of the person who put the entire system together to fulfill the needs of their community per our mission statement. If my mythical llama herders need the oldest Libre Office possible but the newest Rust packaging and whatever random version of httpd that Fedora deems "stable", then that is what I desire, even if the upstream or other non-llama herding users desire something different. However, I'd also push that we should try to reach a point where if a llama herder for non-llama reasons needs a different httpd, they can just enable and use it (using the language of modularity). In case it isn't clear, the "builder's desires" includes the goals of every current lab, spin, and edition, separately and where appropriate together. _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
-- Brian (bex) Exelbierd | bexelbie@redhat.com | bex@pobox.com Fedora Community Action & Impact Coordinator @bexelbie | http://www.winglemeyer.org _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 7:43 AM Fabio Valentini decathorpe@gmail.com wrote:
I think the kernel is a bad example here. It's exceedly stable across releases, so it's probably the one component that's least problematic to upgrade during a fedora release. The fedora kernel team is already doing that, and they are doing an excellent job, which they definitely deserve more praise for than they get. For me, the frequent, well-executed stable kernel updates are one thing that positively distinguishes fedora from other non-rolling distros.
[...snip...]
I wanted to offer a hearty +1 here. I have been thinking more about the optimization needed for a faster release process, which would *not* disadvantage our awesome kernel team.
I wouldn't foresee, for example, that we'd freeze on a specific kernel for such a process. On the contrary, the kernel team does an *amazing* job making sure that Fedora is relevant to the upstream kernel community as much as possible. We routinely get fresh kernels with updated hardware support. I wouldn't want to see this stop.
I understand the lifecycle goals theoretically also make possible a longer term release. How that would happen should be based on what its consumers need, and (maybe even more importantly) what our maintainers are able to do without each being issued a Fedora Time Turner[1]. I don't want to optimize for that case because a much shorter cycle (even with less fanfare) encourages us to pursue more automation and more contributor empowerment.
On Wed, Nov 28, 2018 at 3:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
I think that Fedora's role as an innovater in the OS space means we should be aggressively exploring this. Rolling Releases, Tech-Driven Releases and Time-Based Releases all have well known positives and negatives. All of the work that has been done on Modularity, containers, flatpaks, OSTree, and more, gives us the opportunity to really re-think this. While it is true there are dozens (or more) additional solutions to the too-fast/too-slow and the incompatible-libraries problems, these technologies seem to be gaining the most adoption across a variety of use cases. They are all also generally well supported in Fedora and we need to aggressively push them to achieve this goal or determine where the dead-end is so we can move to the next innovation.
We have a lot of flexibility with all these technologies. What we need to achieve through our processes - CI, the release process, etc, is to enable users to take advantage of the technologies to achieve their goals - to get their work done - without having new problems thrown at them - excessive complexity, instability, etc.
I personal am hugely in favor of us adopting a bootable-base model that allows us to iterate the kernel and the various user-space pieces at the speed of the upstream, the user's desires and the builder's desires[^0] all at the same time. While this will require us to do some level of NxM matrix building and testing, a base that didn't have to change often for most use cases reduces the matrix considerably.
The term "bootable-base" concerns me - because it could be interpreted to mean a minimal bootable set - just enough to get systemd and dnf running, perhaps. In another mail, you explained what you meant as "light up the machine level [...] and get containers/flatpaks/etc running". What does it take to get containers running usefully? It probably looks quite a bit like Fedora CoreOS. What does it take to get Flatpaks running in a useful way to users? - you need a full desktop environment - something like Fedora Silverblue (or the default install set of Fedora Workstation if you prefer loose packages).
While the Fedora release today contains a lot more software than this "core operating system" - generally our release processes today are targeted at trying to release a stable core operating system - and this is something we can't lose if we start changing around how things work. It's not that I don't think the maintenance of glibc and gcc is really important, but if we refocus our release processes only on that, and one day a broken wpa-supplicant rebase lands and half our users lose networking - that would be a really poor outcome for Fedora.
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Fedora needs to be an operating system provider, not just an operating system toolbox provider. There are a lot of use cases for the "operating system toolbox"
- To build the operating systems that Fedora releases - CoreOS, Workstation, etc. - To create the containers, Flatpak runtimes, and Flatpaks that Fedora releases - For users to create application containers - To create development environments (pet containers) - For adventurous users to create new, customized operating systems
but you can't do integration testing on operating system toolbox, because the whole point is that you can integrated it any way you want and only the end user defines what is broken and what is working. Producing a *quality* operating system requires us to focus our processes on the integrated operating systems that we produce, on the applications we produce - and that's what our processes should be around.
Could we replace the current 6 month release processes with an alternate set of processes around multiple rolling branches? It's certainly a possibility, but it's going to be very difficult to get a high level of stability - a level of stability that is acceptable to all end users - without a process where the *entire operating system* gets beta tested before being pushed out.
If we want to move in the direction of rolling releases, the first step is to get the continuous integration testing and gating in place to make rawhide consumable by a broader set of people. That would be incredibly useful even if we keep the current 6 month tempo, but essential if we want to move to a rolling stable model. Until we can demonstrate that, it seems really premature to talk about rolling stable models.
Owen
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
On Wed, Nov 28, 2018 at 3:07 AM Brian (bex) Exelbierd bexelbie@redhat.com wrote:
I'd push Brendans' concept further and suggest that we try to eliminate as many of the compilers, libraries and core system tools as possible from this bootable-base so that those can iterate at their own speed, perhaps 4 year for a laptop vendor and 30 day for a experimental ARM device. Fedora as a project might not build output for the whole range, but a build system that allowed us to help others be successful would be a huge help here.
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with. [Especially during the days when Mandrake and a couple others were just rebuilds of Red Hat Linux with specific kernel flags and some light patching to sed Red Hat Linux to Mandrake.]
Do we even know what an operating system provider is? Because it seems like we do this every release where each groups look at what was produced versus what they wanted and all they can see is all the compromises they had to make to get it out the door. And unless we actually come up with some specific things of what is really wanted, we will be doing this every release in the future too.
On Wed, Nov 28, 2018 at 12:40 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with.
I don't think we've just been saying this, I think we've been steadily improving in this area - both in our focus and in our processes. The move to editions was particularly helpful.
Owen
On Wed, 28 Nov 2018 at 12:47, Owen Taylor otaylor@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:40 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with.
I don't think we've just been saying this, I think we've been steadily improving in this area - both in our focus and in our processes. The move to editions was particularly helpful.
I didn't mean to assume we haven't improved on it, but I do believe it is the central 'conflict' we have had. My main worry is that can we actually achieve being an "operating system provider" or are we always going to be a toolbox who isn't happy with itself? (or thirdly, are we already an operating system provider with delusions of being a toolbox?)
How can we know if we actually are still making progress if we each have a different definition of what that operating system is.
Owen _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 12:57 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 12:47, Owen Taylor otaylor@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:40 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with.
I don't think we've just been saying this, I think we've been steadily improving in this area - both in our focus and in our processes. The move to editions was particularly helpful.
I didn't mean to assume we haven't improved on it, but I do believe it is the central 'conflict' we have had. My main worry is that can we actually achieve being an "operating system provider" or are we always going to be a toolbox who isn't happy with itself? (or thirdly, are we already an operating system provider with delusions of being a toolbox?)
Why do either of those possibilities matter? Or asked more directly, why wouldn't we continue to look at ourselves and iterate and try to improve? There is no "done" state in an operating system.
How can we know if we actually are still making progress if we each have a different definition of what that operating system is.
Great question. I know there are different ideas around it in terms of platform vs. core/extras vs. lifecycles of different bits. I think coming to a general consensus on that is a huge part of the discussion we're having here.
josh
On Wed, 28 Nov 2018 at 13:05, Josh Boyer jwboyer@fedoraproject.org wrote:
On Wed, Nov 28, 2018 at 12:57 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 12:47, Owen Taylor otaylor@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:40 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with.
I don't think we've just been saying this, I think we've been steadily improving in this area - both in our focus and in our processes. The move to editions was particularly helpful.
I didn't mean to assume we haven't improved on it, but I do believe it is the central 'conflict' we have had. My main worry is that can we actually achieve being an "operating system provider" or are we always going to be a toolbox who isn't happy with itself? (or thirdly, are we already an operating system provider with delusions of being a toolbox?)
Why do either of those possibilities matter? Or asked more directly, why wouldn't we continue to look at ourselves and iterate and try to improve? There is no "done" state in an operating system.
You communicated clearer in your previous email what I wanted to say better than I have in mine. Evolution versus revolution. We spend a lot of time and energy talking for and against revolutions every release because we don't seem to know what we are at and when in fact most of them are evolutions which later on weren't as big as expected.
I just want us to come to some consensus on what we have accomplished, where we are at, and then see if we can work out what we need to iterate and improve. [I am probably still not communicating this any better than I did in my previous emails, as I think you covered it very well previously. ]
How can we know if we actually are still making progress if we each have a different definition of what that operating system is.
Great question. I know there are different ideas around it in terms of platform vs. core/extras vs. lifecycles of different bits. I think coming to a general consensus on that is a huge part of the discussion we're having here.
Thanks. I am interested in working out what we have accomplished so far, and what we are hurting on in concrete terms.
josh _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Wed, Nov 28, 2018 at 12:47 PM Owen Taylor otaylor@redhat.com wrote:
On Wed, Nov 28, 2018 at 12:40 PM Stephen John Smoogen smooge@gmail.com wrote:
On Wed, 28 Nov 2018 at 11:37, Owen Taylor otaylor@redhat.com wrote:
Fedora needs to be an operating system provider, not just an operating system toolbox provider. <cut>
I feel like we have been saying this for 15+ years even before Fedora was Fedora. Even back in the RHL days, we would argue over whether what we were providing was an 'OS' or not versus a toolkit for someone else to work with.
I don't think we've just been saying this, I think we've been steadily improving in this area - both in our focus and in our processes. The move to editions was particularly helpful.
I think this is a key observation. Some people thought Editions were just rearranging stuff arbitrarily or for little value. I continue to see them as a great way to focus on specific user bases. What we're discussing now is just continuation of that at a more fundamental level.
For some reason I can't understand, people want a revolution every time. Fedora often has a tendency to look at evolution as failure. That isn't sustainable and leads to repeating mistakes, creating totally new problems, etc. There is nothing wrong with iteration as long as you know where you're going and why you're making the changes you're making.
In this particular round, I think focusing on some of the fundamentals on how we construct our OS (compose tooling, real actual CI, etc) is another evolution. Does it materially change what Fedora is to end users? No. Does it have benefits for them and for maintainers in the long run? Yes.
We seem to also keep trying to force fit a single OS release methodology into all use cases. That leads to a poor fit many times. With looking at multiple lifecycles and how to pull that off using the fundamentals above, we can actually still look at being an OS provider but it doesn't have to be a singular OS. We can create the platform necessary to have commonality at certain layers across different use cases.
Do I think all of this will happen in a single Fedora "release"? No. It's good to be somewhat timeboxed, and I think we can make really great strides on some of it. For other bits we'll have to see how things fall out going forward. But if we don't take the time now to work on the enablers, the big picture or theoretical goals won't matter.
josh
On Tue, Nov 27, 2018 at 12:19 PM Josh Boyer jwboyer@fedoraproject.org wrote:
On Tue, Nov 27, 2018 at 11:21 AM Owen Taylor otaylor@redhat.com wrote:
On Tue, Nov 27, 2018 at 11:05 AM Josh Boyer jwboyer@fedoraproject.org wrote:
[...snip...]
I completely disagree. Our release process and tooling is built on heroism and tech debt. At some point, that is going to cause significant burnout and then people will just wonder what happened when those people stop doing releases. I think it's better to raise awareness at the project level, and build something that is sustainable rather than predicated on "small group of people killing themselves for the greater good". That starts with individual package CI gating, and goes all the way through integration testing between packages in a gated manner, into how we actually *create* all of that plus the things we deem worth of releasing.
I think you are misunderstanding me somehow. I'm 100% on-board with improving the processes, catching compose problems in an automated and early fashion, making developers fix their own problems, and generally making releases a less heroic process. And if that requires a temporary cadence chance to get the retooling done, then it's worth it.
Ah, good!
But Brendan's proposal to permanently slow down the cadence seems to make the assumption that the current release cycle is a heroic effort for the entire project - that we're moving too fast to stay on our feet. And I don't think that's the case.
That would be a concern to me as well, assuming we stuck with a *single* cadence and a *single* platform. With the lifecycle Objective, aren't we targeting multiple cadences and lifecycles? I mean, we have this today with Fedora Atomic/CoreOS vs. "regular" Fedora. Maybe Brendan was thinking singular, but I would very much like the ability in our tooling a process to have both the faster moving cadence of today AND a slower moving one. That's another reason I view a temporary halt as worthwhile. If done correctly, it enables Fedora to fill more usecases and lifecycles than we could ever accomplish with a single release.
Josh and Owen both have it basically right IMHO. Speeding up the compose, increasing automation, and making it easier for people beyond the core crew to do releases all should allow us to make releases less of a big deal. I would love to get closer to the ostree release (both in terms of content, and cycle) for the majority of current users. That means faster, not slower releases -- and that "release" should be more like a non-event for most people. Not to mention the benefits of being able to revert the platform in case of a problem, etc.
But I also agree that a longer cycle can have a place too. The tricky part is to make sure that doesn't turn into multi-stream pain for the proverbial "1000 packagers" group. There should be a way to set the cycle and the overlap to minimize pain. (And I suspect modularity also helps somewhat here.)
Take an average packager who cares about long-term. They may have to maintain 5+ streams now if you count EPEL. We want an outcome that is arguably better, or at least no worse. (No, we really want better.) ;-)
I agree with the folks in this subthread, but I think we are going to have to look at 'redesigning' things more than just 'optimizing'.
ie, collect all our inputs and outputs and things we need to do in the process and figure out how to make it modular (no relation) so we can look at just a single changed input and quickly test and generate all the outputs that are affected by that input (and not any of the others).
For example, a new xfce4-settings package comes out, we want to test it, test the Xfce live media and if it all passes, perhaps release that media.
It may well be that we can figure out a 'base platform' just by looking at these imput/outputs: what things are in most or all outputs?
kevin
On 11/28/18 1:32 AM, Kevin Fenzi wrote:
I agree with the folks in this subthread, but I think we are going to have to look at 'redesigning' things more than just 'optimizing'.
+1. At the same time, we should also ensure that Devel,QE,Infra,RelEng.. all equal stake holders and are hand in glove with the new design and not fall into their own silo.
ie, collect all our inputs and outputs and things we need to do in the process and figure out how to make it modular (no relation) so we can look at just a single changed input and quickly test and generate all the outputs that are affected by that input (and not any of the others).
For example, a new xfce4-settings package comes out, we want to test it, test the Xfce live media and if it all passes, perhaps release that media.
It may well be that we can figure out a 'base platform' just by looking at these imput/outputs: what things are in most or all outputs?
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
I totally agree with Paul and Kevin.
I want to see a faster release cycle (probably rolling release) and shorter processes to get a release out.
On Tue, Nov 27, 2018 at 3:02 PM Kevin Fenzi kevin@scrye.com wrote:
I agree with the folks in this subthread, but I think we are going to have to look at 'redesigning' things more than just 'optimizing'.
ie, collect all our inputs and outputs and things we need to do in the process and figure out how to make it modular (no relation) so we can look at just a single changed input and quickly test and generate all the outputs that are affected by that input (and not any of the others).
For example, a new xfce4-settings package comes out, we want to test it, test the Xfce live media and if it all passes, perhaps release that media.
If we can do this, we can release different artifacts at different times which gives more comfort and flexibility to different WG's.
It may well be that we can figure out a 'base platform' just by looking at these imput/outputs: what things are in most or all outputs?
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Dne 27. 11. 18 v 17:04 Josh Boyer napsal(a):
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt.
People working on release and people working on packages maintenance are different group - they are not disjunct, but it is not the same group. For example *I* am a maintainer of lots of packages, but the additional works because of the fedora release is about one working day per year - and it is mostly because of fedora-upgrade package. Other packages do not need so much work. I am more affected by upstream releases.
Do not forget that annual releases will mean that N-1 release will implicate 24 months support for packages which will mean a much more significant impact on us-the maintaners.
Miroslav
On Thu, Nov 29, 2018 at 4:47 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 27. 11. 18 v 17:04 Josh Boyer napsal(a):
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt.
People working on release and people working on packages maintenance are different group - they are not disjunct, but it is not the same group. For example *I* am a maintainer of lots of packages, but the additional works because of the fedora release is about one working day per year - and it is mostly because of fedora-upgrade package. Other packages do not need so much work. I am more affected by upstream releases.
Do not forget that annual releases will mean that N-1 release will implicate 24 months support for packages which will mean a much more significant impact on us-the maintaners.
Let's not get too focused on an annual release (or any specific timeframe for longer release). I know this is something that some people want. I understand that, and it's *possible* to do in a future state where more people are empowered to make releases happen. But a longer release is not the primary goal, merely something that's possible.
I'm more interested in a shorter release that implies less added maintainer effort. We already put time into Rawhide, and I would like to see that better leveraged in what we push to consumers. Right now our branch cycle is six months, at which point we have numerous freezes and other operations that consume a lot of manual time. They also bottleneck our pace.
I want to increase automation, decrease manual bottlenecks and freezes, and spread out permissions to assemble and push out "ready" content. I would like to optimize for a faster release, making any slower releases possible. Those releases should be based on what the consumers of that release need, as well as the efforts our maintainers have available.
On Thu, Nov 29, 2018 at 10:22 AM Paul Frields stickster@gmail.com wrote:
On Thu, Nov 29, 2018 at 4:47 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 27. 11. 18 v 17:04 Josh Boyer napsal(a):
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt.
People working on release and people working on packages maintenance are different group - they are not disjunct, but it is not the same group. For example *I* am a maintainer of lots of packages, but the additional works because of the fedora release is about one working day per year - and it is mostly because of fedora-upgrade package. Other packages do not need so much work. I am more affected by upstream releases.
Do not forget that annual releases will mean that N-1 release will implicate 24 months support for packages which will mean a much more significant impact on us-the maintaners.
I'll echo what Paul says below with a +1, but I wanted to touch on this point a bit because it implies an assumption that the maintenance model remains the same even if lifecycle options change. I don't think that needs to be the case, nor do I think it would even be good.
Of the large number of packages that you maintain, how many of them are critical to freeze at a specific version for a given Fedora release? Possibly some, but I would think across the distribution it would not be a huge number. So if there is no essential need to freeze them at a specific version, why would you want to maintain the packages *separately* for each release? That sounds like extra work for no benefit. If we instead take a maintenance approach that you maintain package foo and it is built for all releases, then you only really need to maintain it in a singular instance.
Today that is something that can be accomplished with modularity, but I would suggest that we look at stream branching as a solution even for regular packages. So you wouldn't have fc22-fc32 branches under package foo. You'd have foo/stream-<version> and you could build that wherever you'd like. Couple that with automated CI testing and I believe you actually decrease your maintenance burden while increasing your quality.
There are many details that would need to be worked out and I don't want to trivialize them, but I do want to at least get people thinking about it in the long term. If we are going to improve the way we build and deliver our operating system, we shouldn't assume we can do that without changing the way we maintain it either.
josh
Let's not get too focused on an annual release (or any specific timeframe for longer release). I know this is something that some people want. I understand that, and it's *possible* to do in a future state where more people are empowered to make releases happen. But a longer release is not the primary goal, merely something that's possible.
I'm more interested in a shorter release that implies less added maintainer effort. We already put time into Rawhide, and I would like to see that better leveraged in what we push to consumers. Right now our branch cycle is six months, at which point we have numerous freezes and other operations that consume a lot of manual time. They also bottleneck our pace.
I want to increase automation, decrease manual bottlenecks and freezes, and spread out permissions to assemble and push out "ready" content. I would like to optimize for a faster release, making any slower releases possible. Those releases should be based on what the consumers of that release need, as well as the efforts our maintainers have available.
-- Paul _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://getfedora.org/code-of-conduct.html List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
On Thu, Nov 29, 2018 at 10:57 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Nov 29, 2018 at 10:22 AM Paul Frields stickster@gmail.com wrote:
On Thu, Nov 29, 2018 at 4:47 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 27. 11. 18 v 17:04 Josh Boyer napsal(a):
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt.
People working on release and people working on packages maintenance are different group - they are not disjunct, but it is not the same group. For example *I* am a maintainer of lots of packages, but the additional works because of the fedora release is about one working day per year - and it is mostly because of fedora-upgrade package. Other packages do not need so much work. I am more affected by upstream releases.
Do not forget that annual releases will mean that N-1 release will implicate 24 months support for packages which will mean a much more significant impact on us-the maintaners.
I'll echo what Paul says below with a +1, but I wanted to touch on this point a bit because it implies an assumption that the maintenance model remains the same even if lifecycle options change. I don't think that needs to be the case, nor do I think it would even be good.
Of the large number of packages that you maintain, how many of them are critical to freeze at a specific version for a given Fedora release? Possibly some, but I would think across the distribution it would not be a huge number. So if there is no essential need to freeze them at a specific version, why would you want to maintain the packages *separately* for each release? That sounds like extra work for no benefit. If we instead take a maintenance approach that you maintain package foo and it is built for all releases, then you only really need to maintain it in a singular instance.
Today that is something that can be accomplished with modularity, but I would suggest that we look at stream branching as a solution even for regular packages. So you wouldn't have fc22-fc32 branches under package foo. You'd have foo/stream-<version> and you could build that wherever you'd like. Couple that with automated CI testing and I believe you actually decrease your maintenance burden while increasing your quality.
There are many details that would need to be worked out and I don't want to trivialize them, but I do want to at least get people thinking about it in the long term. If we are going to improve the way we build and deliver our operating system, we shouldn't assume we can do that without changing the way we maintain it either.
We can change this _today_, actually. fedpkg supports an in-repo config file to specify distro targets to push whenever running `fedpkg build`. So you could do a repo with only a master branch and have it push to all distro targets enabled for the repo at once. This is probably a useful optimization for the overwhelming majority of packages held by packagers.
It's just not documented or available as an option for setup when you file a repo creation request.
On Thu, Nov 29, 2018 at 11:15 AM Neal Gompa ngompa13@gmail.com wrote:
On Thu, Nov 29, 2018 at 10:57 AM Josh Boyer jwboyer@fedoraproject.org wrote:
On Thu, Nov 29, 2018 at 10:22 AM Paul Frields stickster@gmail.com wrote:
On Thu, Nov 29, 2018 at 4:47 AM Miroslav Suchý msuchy@redhat.com wrote:
Dne 27. 11. 18 v 17:04 Josh Boyer napsal(a):
In other words, the "technical debt" we are trying to solve here is not project wide and doesn't justify slowing down the whole project permanently.
I completely disagree. Our release process and tooling is built on heroism and tech debt.
People working on release and people working on packages maintenance are different group - they are not disjunct, but it is not the same group. For example *I* am a maintainer of lots of packages, but the additional works because of the fedora release is about one working day per year - and it is mostly because of fedora-upgrade package. Other packages do not need so much work. I am more affected by upstream releases.
Do not forget that annual releases will mean that N-1 release will implicate 24 months support for packages which will mean a much more significant impact on us-the maintaners.
I'll echo what Paul says below with a +1, but I wanted to touch on this point a bit because it implies an assumption that the maintenance model remains the same even if lifecycle options change. I don't think that needs to be the case, nor do I think it would even be good.
Of the large number of packages that you maintain, how many of them are critical to freeze at a specific version for a given Fedora release? Possibly some, but I would think across the distribution it would not be a huge number. So if there is no essential need to freeze them at a specific version, why would you want to maintain the packages *separately* for each release? That sounds like extra work for no benefit. If we instead take a maintenance approach that you maintain package foo and it is built for all releases, then you only really need to maintain it in a singular instance.
Today that is something that can be accomplished with modularity, but I would suggest that we look at stream branching as a solution even for regular packages. So you wouldn't have fc22-fc32 branches under package foo. You'd have foo/stream-<version> and you could build that wherever you'd like. Couple that with automated CI testing and I believe you actually decrease your maintenance burden while increasing your quality.
There are many details that would need to be worked out and I don't want to trivialize them, but I do want to at least get people thinking about it in the long term. If we are going to improve the way we build and deliver our operating system, we shouldn't assume we can do that without changing the way we maintain it either.
We can change this _today_, actually. fedpkg supports an in-repo config file to specify distro targets to push whenever running `fedpkg build`. So you could do a repo with only a master branch and have it push to all distro targets enabled for the repo at once. This is probably a useful optimization for the overwhelming majority of packages held by packagers.
Yep, I know :) For a lot of packages, this could be done now. I think to get the full benefits from it across the distro, we'd need to really define that platform layer so maintainers know what they can depend on, etc.
It's just not documented or available as an option for setup when you file a repo creation request.
Right. I think we should start encouraging its use. I kind of dislike using 'master' as the branch because it's ambiguous depending on how you look at it, but it's not wrong. A stream branch just provides a bit more context.
josh
Le jeudi 29 novembre 2018 à 10:56 -0500, Josh Boyer a écrit :
Of the large number of packages that you maintain, how many of them are critical to freeze at a specific version for a given Fedora release? Possibly some, but I would think across the distribution it would not be a huge number.
It's not so much that many packages need freezing, but quite a lot need coordinated rebuilds (mini mass rebuilds). Once thing current cadencing provides is "free" mass rebuilds (you just make sure to commit the key packages before branching and releng will usually mass rebuild all the other things that depend on those key parts for some other reason).
Fix the tooling so those mini mass rebuilds are cheap to setup and are not human packager intensive, and there's not reason the rebuild result could not be pushed to every release. Or to a rolling release. Or whatever.
We really need to evolve our tooling from "packages can be handled independently in reviews, builds, and pushes, coordinated changes are the exception" to "we manage sets of packages by default, single-package changes are just set changes with set = 1"
On Thu, Nov 29, 2018 at 12:42 PM Nicolas Mailhot nicolas.mailhot@laposte.net wrote:
Le jeudi 29 novembre 2018 à 10:56 -0500, Josh Boyer a écrit :
Of the large number of packages that you maintain, how many of them are critical to freeze at a specific version for a given Fedora release? Possibly some, but I would think across the distribution it would not be a huge number.
It's not so much that many packages need freezing, but quite a lot need coordinated rebuilds (mini mass rebuilds). Once thing current cadencing provides is "free" mass rebuilds (you just make sure to commit the key packages before branching and releng will usually mass rebuild all the other things that depend on those key parts for some other reason).
Fix the tooling so those mini mass rebuilds are cheap to setup and are not human packager intensive, and there's not reason the rebuild result could not be pushed to every release. Or to a rolling release. Or whatever.
Yes, agreed. Fixing the tooling is what much of these threads are about ;)
We really need to evolve our tooling from "packages can be handled independently in reviews, builds, and pushes, coordinated changes are the exception" to "we manage sets of packages by default, single-package changes are just set changes with set = 1"
Modularity does that for a number of cases, but it isn't a silver bullet. Our package dependencies web is much too complex to make it feasible for everything to be in a module, at least right now.
(We should really also revisit our package dependencies in general and try to minimize as much as we can for a variety of reasons, leveraging rich/weak deps. That's a different topic though.)
josh
-
Adam Williamson -
Ben Cotton -
Brendan Conoboy -
Brian (bex) Exelbierd -
Clement Verna -
Dennis Gilmore -
drago01 -
Fabio Valentini -
Igor Gnatenko -
Japheth Cleaver -
John Florian -
Josh Boyer -
Justin Forbes -
Kevin Fenzi -
Kevin Kofler -
Matthew Miller -
mcatanzaro＠gnome.org -
Miroslav Suchý -
Mohan Boddu -
Neal Gompa -
Nicolas Mailhot -
Owen Taylor -
Paul Frields -
Peter Robinson -
Ralf Corsepius -
Randy Barlow -
Stephen Gallagher -
Stephen John Smoogen -
Sudhir -
Vít Ondruch