For a *very* long time, Rawhide has built rcX kernels as "release" kernels and daily git snapshots as debug kernels only. This has brought attention to some issues that might otherwise be missed. Specifically around things like lockdep. Unfortunately, even without changing our selected debug options, performance has gotten considerably worse for these debug kernels due to upstream code changes. At the same time, we do have some additional automated testing happening now that was not happening before, which we can explicitly point at debug kernels. I am wondering if it is time to switch rawhide to work the way that everything else does, building both debug and nodebug builds every day, instead of forcing debug builds 80% of the time. It would mean a reduction in coverage, but I am not sure by how much, as I get the impression that most rawhide users are sticking to the nodebug kernels anyway. It would also allow us to turn on some more debug features for our debug kernels that are a bit more performance limiting, but extremely useful. We have left them off in Fedora specifically because they were too heavy weight for expecting users to run them.
So, what does the community think? Should we continue as is, or should we move to a more typical model where both debug and non-debug kernels are build with every build?
Thanks, Justin
On Wed, Jan 18, 2023 at 12:04:41 -0600, Justin Forbes jforbes@fedoraproject.org wrote:
So, what does the community think? Should we continue as is, or should we move to a more typical model where both debug and non-debug kernels are build with every build?
It would be slightly more convenient for me to have daily nodebug kernels. I probably would reboot more often to use them instead of weekly or when I reboot for some reason other than to move to the latest nodebug kernel. I use some older hardware so debug kernels may be more of an issue for me than others.
Probably what makes sense is what is easier for you. You're a limited resource and saving time spent doing builds so that you can look at other things, has benefit that also affects people.
On Wed, Jan 18, 2023 at 12:04:41PM -0600, Justin Forbes wrote: ...snip....
So, what does the community think? Should we continue as is, or should we move to a more typical model where both debug and non-debug kernels are build with every build?
So yeah, I used to reboot pretty much daily and use the new kernel every day, but a while back due to the performance problems you indicated I have gone to just updating to the nodebug rc's on mondays and keeping that until the next week.
If daily ones were nodebug I would go back to rebooting mostly daily, so you would get more test coverage from me at least. ;)
Additionally, I think the debug kernels are a bit bigger and sometimes that causes problems with images, as well as sometimes images take longer to build with the debug version in use.
so, +1 from me to switch to nodebug all the time.
kevin
Agree; daily nodebugs would be useful and I would likely switch back to rawhide kernels.
I tend to move my daily systems to userspace rawhide once the -next release tag from the GA release, but have often had to run copr based non FC kernels for various things when doing so, which is less than ideal.
On Thu, 19 Jan 2023 at 11:18, Kevin Fenzi kevin@scrye.com wrote:
On Wed, Jan 18, 2023 at 12:04:41PM -0600, Justin Forbes wrote: ...snip....
So, what does the community think? Should we continue as is, or should we move to a more typical model where both debug and non-debug kernels are build with every build?
So yeah, I used to reboot pretty much daily and use the new kernel every day, but a while back due to the performance problems you indicated I have gone to just updating to the nodebug rc's on mondays and keeping that until the next week.
If daily ones were nodebug I would go back to rebooting mostly daily, so you would get more test coverage from me at least. ;)
Additionally, I think the debug kernels are a bit bigger and sometimes that causes problems with images, as well as sometimes images take longer to build with the debug version in use.
so, +1 from me to switch to nodebug all the time.
kevin _______________________________________________ kernel mailing list -- kernel@lists.fedoraproject.org To unsubscribe send an email to kernel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/kernel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Wed, Jan 18, 2023 at 12:04:41PM -0600, Justin Forbes wrote:
For a *very* long time, Rawhide has built rcX kernels as "release" kernels and daily git snapshots as debug kernels only. This has brought attention to some issues that might otherwise be missed. Specifically around things like lockdep. Unfortunately, even without changing our selected debug options, performance has gotten considerably worse for these debug kernels due to upstream code changes. At the same time, we do have some additional automated testing happening now that was not happening before, which we can explicitly point at debug kernels. I am wondering if it is time to switch rawhide to work the way that everything else does, building both debug and nodebug builds every day, instead of forcing debug builds 80% of the time. It would mean a reduction in coverage, but I am not sure by how much, as I get the impression that most rawhide users are sticking to the nodebug kernels anyway. It would also allow us to turn on some more debug features for our debug kernels that are a bit more performance limiting, but extremely useful. We have left them off in Fedora specifically because they were too heavy weight for expecting users to run them.
So, what does the community think? Should we continue as is, or should we move to a more typical model where both debug and non-debug kernels are build with every build?
If Rawhide is to maximise its usefulness as a distro you can actually run tests on, then its functional behaviour and operation needs to be on a par with the final released distro will have. With the debug kernels provided by default for most of the rawhide cycle, this goal is not achievable today.
Having debug kernels available for testing is a good idea, and we could encourage users to try them. I think it is undesirable to force the debug builds onto every install by default though, because their operational behaviour (mostly in terms of performance) is not a match for what will actually be shipped.
With regards, Daniel
I'm not too great at figuring how you tell which is which from the build process, but there was a comment in a kernel build today that suggests that things have changed so that now rawhide kernels are nodebug by default. Is that correct?
On Thu, Jan 26, 2023 at 4:39 PM Bruno Wolff III bruno@wolff.to wrote:
I'm not too great at figuring how you tell which is which from the build process, but there was a comment in a kernel build today that suggests that things have changed so that now rawhide kernels are nodebug by default. Is that correct?
Yes, that was the result of this discussion tied with some bugzilla discussion and other various places. As the MR that did it is "include in release" it has been the case for all RC5 and newer builds, but I was waiting to make an official announcement until the MR is approved and merged.
Justin
kernel@lists.fedoraproject.org
-
Bruno Wolff III -
Daniel P. Berrangé -
Joel Wirāmu Pauling -
Justin Forbes -
Kevin Fenzi