Hi folks,
Today something happened, that happens over and over again with Fedora, and it makes me angry. I am running Fedora 17, and so far it worked well with the initial kernel 3.3.x (except that it would panic on shutdown... but that was not important to me, but still embarassing). Today I was notified of an important security update in the kernel. Curiously, it would update from 3.3 to 3.4 (a major version upgrade, which should not happen in such a core package anyway, IMO). Reboot into the new kernel, everything comes up --- until I want to actually want to read email, surf web, or anything that requires my network. I am on an Intel Wifi card, iwlwifi module. I *can* connect to the network, but everything is suuuuuuper slow or times-out every now and then. Completely unusable. Reboot into the older kernel, things work well again. Now I am left with the choice of running a new kernel w/o network or an unsecure kernel. Thank you very much!
This sort of thing I would expect in rawhide/development builds, but not in a supposed-to-be stable release. I can understand the underlying idea of being on the bleeding edge, but I don't want to actually be bleeding. At least the base system components should not undergo major version updates. Security fixes should be backported to the software version that is in the stable release (1 year release cycle shouldn't be too demanding for this), and only security fixes and absolutely important fixes should go into stable releases. (Not to mention that some fixes that I *would* consider important enough to go into stable never end up there.) If major version updates are really really necessary, they should undergo serious testing. I cannot believe that I am the only one on an Intel Wifi chip. The way it is now, Fedora feels like a constantly rolling development version that is almost unusable (because any update, even security, has a fairly high risk of breaking things) for day-to-day work.
Bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=831571
Best regards, Roman
Today something happened, that happens over and over again with Fedora, and it makes me angry. I am running Fedora 17, and so far it worked well with the initial kernel 3.3.x (except that it would panic on shutdown... but that was not important to me, but still embarassing). Today I was notified of an important security update in the kernel. Curiously, it would update from 3.3 to 3.4 (a major version upgrade, which should not happen in such a core package anyway, IMO). Reboot into the new kernel, everything comes up --- until I want to actually want to read email, surf web, or anything that requires my network. I am on an Intel Wifi card, iwlwifi module. I *can* connect to the network, but everything is suuuuuuper slow or times-out every now and then. Completely unusable. Reboot into the older kernel, things work well again. Now I am left with the choice of running a new kernel w/o network or an unsecure kernel. Thank you very much!
This sort of thing I would expect in rawhide/development builds, but not in a supposed-to-be stable release. I can understand the underlying idea of being on the bleeding edge, but I don't want to actually be bleeding. At least the base system components should not undergo major version updates. Security fixes should be backported to the software version that is in the stable release (1 year release cycle shouldn't be too demanding for this), and only security fixes and absolutely important fixes should go into stable releases. (Not to mention that some fixes that I *would* consider important enough to go into stable never end up there.) If major version updates are really really necessary, they should undergo serious testing. I cannot believe that I am the only one on an Intel Wifi chip. The way it is now, Fedora feels like a constantly rolling development version that is almost unusable (because any update, even security, has a fairly high risk of breaking things) for day-to-day work.
Bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=831571
Since I just received an email in private pointing out that emails like mine above might be discouraging and not helpful... let me apologize for this. My intention is not to bash other people's best efforts, but instead try to help out (otherwise I would not bother to diligently file bugreports and mention my concerns on this list). I am willing to help track down and fix the problem. However, I see a more general problem and maybe we can turn this into a discussion how to address (or answer) it.
- Why do we allow new major versions of core components into a stable release? What sort of testing is performed before a major kernel update hits Fedora stable? - What is the policy with regards to risky changes (like unnecessary feature updates, ABI changes, etc) in stable? - How can problems like the one I described above be avoided? Is there anything I and others can help with?
Roman
On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke rkennke@redhat.com wrote:
Today something happened, that happens over and over again with Fedora, and it makes me angry. I am running Fedora 17, and so far it worked well with the initial kernel 3.3.x (except that it would panic on shutdown... but that was not important to me, but still embarassing). Today I was notified of an important security update in the kernel. Curiously, it would update from 3.3 to 3.4 (a major version upgrade, which should not happen in such a core package anyway, IMO). Reboot into the new kernel, everything comes up --- until I want to actually want to read email, surf web, or anything that requires my network. I am on an Intel Wifi card, iwlwifi module. I *can* connect to the network, but everything is suuuuuuper slow or times-out every now and then. Completely unusable. Reboot into the older kernel, things work well again. Now I am left with the choice of running a new kernel w/o network or an unsecure kernel. Thank you very much!
This sort of thing I would expect in rawhide/development builds, but not in a supposed-to-be stable release. I can understand the underlying idea of being on the bleeding edge, but I don't want to actually be bleeding. At least the base system components should not undergo major version updates. Security fixes should be backported to the software version that is in the stable release (1 year release cycle shouldn't be too demanding for this), and only security fixes and absolutely important fixes should go into stable releases. (Not to mention that some fixes that I *would* consider important enough to go into stable never end up there.) If major version updates are really really necessary, they should undergo serious testing. I cannot believe that I am the only one on an Intel Wifi chip. The way it is now, Fedora feels like a constantly rolling development version that is almost unusable (because any update, even security, has a fairly high risk of breaking things) for day-to-day work.
Bugzilla report: https://bugzilla.redhat.com/show_bug.cgi?id=831571
Since I just received an email in private pointing out that emails like mine above might be discouraging and not helpful... let me apologize for this. My intention is not to bash other people's best efforts, but instead try to help out (otherwise I would not bother to diligently file bugreports and mention my concerns on this list). I am willing to help track down and fix the problem. However, I see a more general problem and maybe we can turn this into a discussion how to address (or answer) it.
- Why do we allow new major versions of core components into a stable
release? What sort of testing is performed before a major kernel update hits Fedora stable?
- What is the policy with regards to risky changes (like unnecessary
feature updates, ABI changes, etc) in stable?
- How can problems like the one I described above be avoided? Is there
anything I and others can help with?
Roman
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Johannes
Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke rkennke@redhat.com wrote: > Today something happened, that happens over and over again with Fedora, > and it makes me angry. I am running Fedora 17, and so far it worked well > with the initial kernel 3.3.x (except that it would panic on shutdown... > but that was not important to me, but still embarassing). Today I was > notified of an important security update in the kernel. Curiously, it > would update from 3.3 to 3.4 (a major version upgrade, which should not > happen in such a core package anyway, IMO). Reboot into the new kernel, > everything comes up --- until I want to actually want to read email, > surf web, or anything that requires my network. I am on an Intel Wifi > card, iwlwifi module. I *can* connect to the network, but everything is > suuuuuuper slow or times-out every now and then. Completely unusable. > Reboot into the older kernel, things work well again. Now I am left with > the choice of running a new kernel w/o network or an unsecure kernel. > Thank you very much! > > This sort of thing I would expect in rawhide/development builds, but not > in a supposed-to-be stable release. I can understand the underlying idea > of being on the bleeding edge, but I don't want to actually be bleeding. > At least the base system components should not undergo major version > updates. Security fixes should be backported to the software version > that is in the stable release (1 year release cycle shouldn't be too > demanding for this), and only security fixes and absolutely important > fixes should go into stable releases. (Not to mention that some fixes > that I *would* consider important enough to go into stable never end up > there.) If major version updates are really really necessary, they > should undergo serious testing. I cannot believe that I am the only one > on an Intel Wifi chip. The way it is now, Fedora feels like a constantly > rolling development version that is almost unusable (because any update, > even security, has a fairly high risk of breaking things) for day-to-day > work. > > Bugzilla report: > https://bugzilla.redhat.com/show_bug.cgi?id=831571
Since I just received an email in private pointing out that emails like mine above might be discouraging and not helpful... let me apologize for this. My intention is not to bash other people's best efforts, but instead try to help out (otherwise I would not bother to diligently file bugreports and mention my concerns on this list). I am willing to help track down and fix the problem. However, I see a more general problem and maybe we can turn this into a discussion how to address (or answer) it. - Why do we allow new major versions of core components into a stable release? What sort of testing is performed before a major kernel update hits Fedora stable? - What is the policy with regards to risky changes (like unnecessary feature updates, ABI changes, etc) in stable? - How can problems like the one I described above be avoided? Is there anything I and others can help with? Roman -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Roman
Hi,
On Wed, 13 Jun 2012 14:15:14 +0200 Roman Kennke rkennke@redhat.com wrote:
Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Check
https://admin.fedoraproject.org/updates/kernel
for updates. Provide negative karma for them in Bodhi as well:
http://fedoraproject.org/wiki/QA/Updates_Testing
Regards,
--Stijn
Am Mittwoch, den 13.06.2012, 14:29 +0200 schrieb Stijn Hoop:
Hi,
On Wed, 13 Jun 2012 14:15:14 +0200 Roman Kennke rkennke@redhat.com wrote:
Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Check
https://admin.fedoraproject.org/updates/kernel
for updates. Provide negative karma for them in Bodhi as well:
Thanks everybody!
I subscribed to the above updates/kernel RSS feed and will try to test kernels before they go to stable.
Would it make sense to require more karma than just the default 3? Looking at:
https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
I see that there are 5 oks and 2 denys, which actually point to bug reports, both sound fairly important. How does the karma system work if, e.g., update requires +3, the update gets +4 and -1, and this -1 is something that can be considered a release critical bug? data corruption sounds quite release-critical? Is there a mechanism that prevents the update to happen in this case?
Roman
On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke rkennke@redhat.com wrote:
Would it make sense to require more karma than just the default 3? Looking at:
https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
I see that there are 5 oks and 2 denys, which actually point to bug reports, both sound fairly important. How does the karma system work if, e.g., update requires +3, the update gets +4 and -1, and this -1 is something that can be considered a release critical bug? data corruption sounds quite release-critical? Is there a mechanism that prevents the update to happen in this case?
Good questions.
The person that submits the update gets emails for every comment added to the update. This particular one had a couple things that happened though.
1) It got the requisite karma for stable rather quickly 2) Justin was on vacation when the negative karma was submitted. Bodhi only emails the update submitter and the rest of the kernel team didn't notice.
I'm sure that it would have been pulled if Justin was actually around or if the rest of the kernel team had remembered to go check the update. It's something that can be looked at in the future.
josh
On Wed, 2012-06-13 at 09:36 -0400, Josh Boyer wrote:
On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke rkennke@redhat.com wrote:
Would it make sense to require more karma than just the default 3? Looking at:
https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
I see that there are 5 oks and 2 denys, which actually point to bug reports, both sound fairly important. How does the karma system work if, e.g., update requires +3, the update gets +4 and -1, and this -1 is something that can be considered a release critical bug? data corruption sounds quite release-critical? Is there a mechanism that prevents the update to happen in this case?
Good questions.
The person that submits the update gets emails for every comment added to the update. This particular one had a couple things that happened though.
- It got the requisite karma for stable rather quickly
- Justin was on vacation when the negative karma was submitted. Bodhi
only emails the update submitter and the rest of the kernel team didn't notice.
I'm sure that it would have been pulled if Justin was actually around or if the rest of the kernel team had remembered to go check the update. It's something that can be looked at in the future.
This can also serve as my quarterly reminder that Bodhi 2.0 is _still_ supposed to be coming, with much better feedback features. The simple +/- points system in Bodhi 1.0 isn't really adequate for any number of scenarios, including this one. I have posted before about what benefits a better feedback system would have: https://lists.fedoraproject.org/pipermail/devel/2011-November/159874.html
Am Mittwoch, den 13.06.2012, 12:07 -0700 schrieb Adam Williamson:
On Wed, 2012-06-13 at 09:36 -0400, Josh Boyer wrote:
On Wed, Jun 13, 2012 at 9:24 AM, Roman Kennke rkennke@redhat.com wrote:
Would it make sense to require more karma than just the default 3? Looking at:
https://admin.fedoraproject.org/updates/FEDORA-2012-8824/kernel-3.4.0-1.fc17
I see that there are 5 oks and 2 denys, which actually point to bug reports, both sound fairly important. How does the karma system work if, e.g., update requires +3, the update gets +4 and -1, and this -1 is something that can be considered a release critical bug? data corruption sounds quite release-critical? Is there a mechanism that prevents the update to happen in this case?
Good questions.
The person that submits the update gets emails for every comment added to the update. This particular one had a couple things that happened though.
- It got the requisite karma for stable rather quickly
- Justin was on vacation when the negative karma was submitted. Bodhi
only emails the update submitter and the rest of the kernel team didn't notice.
I'm sure that it would have been pulled if Justin was actually around or if the rest of the kernel team had remembered to go check the update. It's something that can be looked at in the future.
This can also serve as my quarterly reminder that Bodhi 2.0 is _still_ supposed to be coming, with much better feedback features. The simple +/- points system in Bodhi 1.0 isn't really adequate for any number of scenarios, including this one. I have posted before about what benefits a better feedback system would have: https://lists.fedoraproject.org/pipermail/devel/2011-November/159874.html
What you lay out there sounds useful indeed! I hope it's coming.
One thing that might be useful to think about is the way Debian handles transitions. (Forgive me if the following is not 100% correct, I am only remotely familiar with what I am saying.) Packages in testing (or unstable) can only transition to the next more stable repository if certain conditions are fulfilled. One is to be the at least X days, another is to not have any (new) release critical bugs in it. (I think there are more, but those two seem important.) In order to accomplish this, the package update system (I guess it would be comparable to Fedora's Bodhi) is linked to the bug DB, and bugs for a package can have a severity of 'release critical' which basically blocks transitions of the package. I think something similar could be useful for Fedora as well, i.e. link Bodhi to bugzilla and react on certain things like severity of bugs.
Of course, it still means that somebody actually needs to test it in order to find and file bugs...
Roman
Josh Boyer wrote:
The person that submits the update gets emails for every comment added to the update. This particular one had a couple things that happened though.
- It got the requisite karma for stable rather quickly
- Justin was on vacation when the negative karma was submitted. Bodhi
only emails the update submitter and the rest of the kernel team didn't notice.
I'm sure that it would have been pulled if Justin was actually around or if the rest of the kernel team had remembered to go check the update. It's something that can be looked at in the future.
Why is karma automatism being enabled (by the submitters, it's optional!) on kernel updates in the first place? IMHO, kernels should always be pushed manually. (I actually consider karma automatism entirely broken, but the kernel is one of the worst packages to use it on!)
Kevin Kofler
On Wed, Jun 13, 2012 at 8:15 AM, Roman Kennke rkennke@redhat.com wrote:
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new
Yes, testing would be very much appreciated.
kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
The kernel maintainers push all updates through updates-testing for this very reason. It is very rare that one is pushed directly to stable. You can use 'yum update --enablerepo=updates-testing kernel' (or similar commands) to only update the kernel from updates testing if you wish. Then you provide karma in Bodhi.
josh
On 13/06/12 13:15, Roman Kennke wrote:
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc)
Test the kernels when they arrive in testing-updates. Well in advance of stable. Give\Deny karma as appropriate. based on what problems may occur.
----- Original Message -----
From: "Roman Kennke" rkennke@redhat.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Wednesday, June 13, 2012 3:15:14 PM Subject: Re: Important kernel update should not break stuff
Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke rkennke@redhat.com wrote: > Today something happened, that happens over and over > again with Fedora, > and it makes me angry. I am running Fedora 17, and so far > it worked well > with the initial kernel 3.3.x (except that it would panic > on shutdown... > but that was not important to me, but still embarassing). Today I was > notified of an important security update in the kernel. Curiously, it > would update from 3.3 to 3.4 (a major version upgrade, > which should not > happen in such a core package anyway, IMO). Reboot into > the new kernel, > everything comes up --- until I want to actually want to read email, > surf web, or anything that requires my network. I am on > an Intel Wifi > card, iwlwifi module. I *can* connect to the network, but everything is > suuuuuuper slow or times-out every now and then. > Completely unusable. > Reboot into the older kernel, things work well again. Now > I am left with > the choice of running a new kernel w/o network or an unsecure kernel. > Thank you very much! > > This sort of thing I would expect in rawhide/development builds, but not > in a supposed-to-be stable release. I can understand the underlying idea > of being on the bleeding edge, but I don't want to > actually be bleeding. > At least the base system components should not undergo > major version > updates. Security fixes should be backported to the > software version > that is in the stable release (1 year release cycle shouldn't be too > demanding for this), and only security fixes and > absolutely important > fixes should go into stable releases. (Not to mention > that some fixes > that I *would* consider important enough to go into > stable never end up > there.) If major version updates are really really necessary, they > should undergo serious testing. I cannot believe that I > am the only one > on an Intel Wifi chip. The way it is now, Fedora feels > like a constantly > rolling development version that is almost unusable > (because any update, > even security, has a fairly high risk of breaking things) for day-to-day > work. > > Bugzilla report: > https://bugzilla.redhat.com/show_bug.cgi?id=831571
Since I just received an email in private pointing out that emails like mine above might be discouraging and not helpful... let me apologize for this. My intention is not to bash other people's best efforts, but instead try to help out (otherwise I would not bother to diligently file bugreports and mention my concerns on this list). I am willing to help track down and fix the problem. However, I see a more general problem and maybe we can turn this into a discussion how to address (or answer) it. - Why do we allow new major versions of core components into a stable release? What sort of testing is performed before a major kernel update hits Fedora stable? - What is the policy with regards to risky changes (like unnecessary feature updates, ABI changes, etc) in stable? - How can problems like the one I described above be avoided? Is there anything I and others can help with? Roman -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Try having updates-testing repo enabled test and provide karma via bodhi. +1 is as needed as -1 as without it we never know whether people simply installed and it worked so they stopped at that point. See http://fedoraproject.org/wiki/Bodhi for details. There is also fedora-easy-karma to ease mass reporting via bodhi. Note that this would not make it easier for you short term as you would spot the problems just a bit earlier. But if enough people do that problems might even be spotted before you update next time and the build untagged so you never see the broken build/update at all. This kind of workflow is effective only if we manage to gather critical mass.
Regards, Alex
Roman
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On 13 June 2012 13:31, Aleksandar Kurtakov akurtako@redhat.com wrote:
----- Original Message -----
From: "Roman Kennke" rkennke@redhat.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Wednesday, June 13, 2012 3:15:14 PM Subject: Re: Important kernel update should not break stuff
- How can problems like the one I described above be avoided? Is there anything I and others can help with?
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Try having updates-testing repo enabled test and provide karma via bodhi. +1 is as needed as -1 as without it we never know whether people simply installed and it worked so they stopped at that point. See http://fedoraproject.org/wiki/Bodhi for details. There is also fedora-easy-karma to ease mass reporting via bodhi. Note that this would not make it easier for you short term as you would spot the problems just a bit earlier. But if enough people do that problems might even be spotted before you update next time and the build untagged so you never see the broken build/update at all. This kind of workflow is effective only if we manage to gather critical mass.
I get the need for people to volunteer and test and that's fine. But the thing I can't square here is why then we aren't all on updates-testing all the time? The kernel is one of the few packages you can guarantee everyone is using. We'd all like to know if an upcoming kernel has problems that will affect us, but you can't know that until you run it.
So the answer is to test it by being on updates-testing, but this means you put yourself in the way of every possible kernel update and open yourself to more potential problems. I don't have a solution for that, but it's clearly something that needs some thought (as someone who has quite a few times been stuck on old kernels and trying updates-testing ones to see if they fix a newly introduced problem).
One question is why are regressions quite so common for some components? iwlwifi for example seems to show up quite a bit.
On Wed, Jun 13, 2012 at 17:27:03 +0100, Ian Malone ibmalone@gmail.com wrote:
I get the need for people to volunteer and test and that's fine. But the thing I can't square here is why then we aren't all on updates-testing all the time? The kernel is one of the few packages you can guarantee everyone is using. We'd all like to know if an upcoming kernel has problems that will affect us, but you can't know that until you run it.
So the answer is to test it by being on updates-testing, but this means you put yourself in the way of every possible kernel update and open yourself to more potential problems. I don't have a solution for that, but it's clearly something that needs some thought (as someone who has quite a few times been stuck on old kernels and trying updates-testing ones to see if they fix a newly introduced problem).
Different people may be more comfortable with dealing with kernel problems then others. This can vary based on the persons skills and how they are using the affected machines.
Part of my contribution to Fedora is being a guinea pig using updates-testing and rawhide. However, dealing with these issues occasionally sucks up a lot of time and I can certainly see why others would need to not do this.
----- Original Message -----
From: "Ian Malone" ibmalone@gmail.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Wednesday, June 13, 2012 7:27:03 PM Subject: Re: Important kernel update should not break stuff
On 13 June 2012 13:31, Aleksandar Kurtakov akurtako@redhat.com wrote:
----- Original Message -----
From: "Roman Kennke" rkennke@redhat.com To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Sent: Wednesday, June 13, 2012 3:15:14 PM Subject: Re: Important kernel update should not break stuff
- How can problems like the one I described above be avoided? Is there anything I and others can help with?
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
Try having updates-testing repo enabled test and provide karma via bodhi. +1 is as needed as -1 as without it we never know whether people simply installed and it worked so they stopped at that point. See http://fedoraproject.org/wiki/Bodhi for details. There is also fedora-easy-karma to ease mass reporting via bodhi. Note that this would not make it easier for you short term as you would spot the problems just a bit earlier. But if enough people do that problems might even be spotted before you update next time and the build untagged so you never see the broken build/update at all. This kind of workflow is effective only if we manage to gather critical mass.
I get the need for people to volunteer and test and that's fine. But the thing I can't square here is why then we aren't all on updates-testing all the time? The kernel is one of the few packages you can guarantee everyone is using. We'd all like to know if an upcoming kernel has problems that will affect us, but you can't know that until you run it.
If we means all people subscribed to fedora-devel I think everyone should run with updates-testing enabled. This definition of we is the people that make Fedora happen so we should always test our stuff - non-stop. Regular updates are for users. From my POV everyone sending messages to fedora-devel is supposed to run with updates-testing enabled. And excuses like "no time" and etc. doesn't make sense as this case is showing that one can not simply care about his packages only because there is always something else that your package collaborates with that needs testing too.
Alex
So the answer is to test it by being on updates-testing, but this means you put yourself in the way of every possible kernel update and open yourself to more potential problems. I don't have a solution for that, but it's clearly something that needs some thought (as someone who has quite a few times been stuck on old kernels and trying updates-testing ones to see if they fix a newly introduced problem).
One question is why are regressions quite so common for some components? iwlwifi for example seems to show up quite a bit.
-- imalone http://ibmalone.blogspot.co.uk -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
On Wed, Jun 13, 2012 at 13:16:24 -0400, Aleksandar Kurtakov akurtako@redhat.com wrote:
If we means all people subscribed to fedora-devel I think everyone should run with updates-testing enabled. This definition of we is the people that make Fedora happen so we should always test our stuff - non-stop. Regular updates are for users. From my POV everyone sending messages to fedora-devel is supposed to run with updates-testing enabled. And excuses like "no time" and etc. doesn't make sense as this case is showing that one can not simply care about his packages only because there is always something else that your package collaborates with that needs testing too.
While I would enourage Fedora contributors to run with updates-testing available, I don't think everyone on the devel list needs to. We all have to make trade offs of time and the project may benefit more from some people spending their time on other things than testing updates before they get to stable. Note that not everyone subscribed to the devel list is a packager and that packagers have other ways to install their package updates without using the updates-testing repo.
Roman Kennke rkennke@redhat.com writes:
Am Mittwoch, den 13.06.2012, 13:05 +0100 schrieb Johannes Lips:
On Wed, Jun 13, 2012 at 1:00 PM, Roman Kennke rkennke@redhat.com wrote: > Today something happened, that happens over and over again with Fedora, > and it makes me angry. I am running Fedora 17, and so far it worked well > with the initial kernel 3.3.x (except that it would panic on shutdown... > but that was not important to me, but still embarassing). Today I was > notified of an important security update in the kernel. Curiously, it > would update from 3.3 to 3.4 (a major version upgrade, which should not > happen in such a core package anyway, IMO). Reboot into the new kernel, > everything comes up --- until I want to actually want to read email, > surf web, or anything that requires my network. I am on an Intel Wifi > card, iwlwifi module. I *can* connect to the network, but everything is > suuuuuuper slow or times-out every now and then. Completely unusable. > Reboot into the older kernel, things work well again. Now I am left with > the choice of running a new kernel w/o network or an unsecure kernel. > Thank you very much! > > This sort of thing I would expect in rawhide/development builds, but not > in a supposed-to-be stable release. I can understand the underlying idea > of being on the bleeding edge, but I don't want to actually be bleeding. > At least the base system components should not undergo major version > updates. Security fixes should be backported to the software version > that is in the stable release (1 year release cycle shouldn't be too > demanding for this), and only security fixes and absolutely important > fixes should go into stable releases. (Not to mention that some fixes > that I *would* consider important enough to go into stable never end up > there.) If major version updates are really really necessary, they > should undergo serious testing. I cannot believe that I am the only one > on an Intel Wifi chip. The way it is now, Fedora feels like a constantly > rolling development version that is almost unusable (because any update, > even security, has a fairly high risk of breaking things) for day-to-day > work. > > Bugzilla report: > https://bugzilla.redhat.com/show_bug.cgi?id=831571
Since I just received an email in private pointing out that emails like mine above might be discouraging and not helpful... let me apologize for this. My intention is not to bash other people's best efforts, but instead try to help out (otherwise I would not bother to diligently file bugreports and mention my concerns on this list). I am willing to help track down and fix the problem. However, I see a more general problem and maybe we can turn this into a discussion how to address (or answer) it. - Why do we allow new major versions of core components into a stable release? What sort of testing is performed before a major kernel update hits Fedora stable? - What is the policy with regards to risky changes (like unnecessary feature updates, ABI changes, etc) in stable? - How can problems like the one I described above be avoided? Is there anything I and others can help with? Roman -- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
I think the reason for shipping the latest upstream kernel is based on the fact that backporting would be too much work. http://fedoraproject.org/wiki/KernelRebases Gives a good overview and probably prevents us from repeating arguments in the discussion.
Ok, fair enough. The question remains, how can we avoid such bad things to happen in the future? Should I regularily try out kernel builds on their way to stable, and object to their stable-release when I find a problem? And how would I do that? (I.e. how can I find out when a new kernel is about to go to stable, and when to test it, etc) And what about the other base components of the system? (Although, to be fair, the kernel seems to be the most problematic one..)
imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0 for f17
On Wed, Jun 13, 2012 at 14:49:25 +0200, Nikola Pajkovsky npajkovs@redhat.com wrote:
imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0 for f17
3.4.2 is available, though the last I checked the build hadn't been pushed to testing. I don't know if that is an oversight or if they didn't feel the kernel was suitable for releasing.
The 3.4 kernels were being tested in rawhide before it was released to f17 and at least of us weren't seeing any regressions with them. So there was reason to believe they would work for people without problems.
On Wed, Jun 13, 2012 at 8:53 AM, Bruno Wolff III bruno@wolff.to wrote:
On Wed, Jun 13, 2012 at 14:49:25 +0200, Nikola Pajkovsky npajkovs@redhat.com wrote:
imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0 for f17
3.4.2 is available, though the last I checked the build hadn't been pushed to testing. I don't know if that is an oversight or if they didn't feel the kernel was suitable for releasing.
I believe it's a matter of getting caught up. There are at least 2 more patches added after the 3.4.2 build that should probably get included in the next update.
The 3.4 kernels were being tested in rawhide before it was released to f17 and at least of us weren't seeing any regressions with them. So there was reason to believe they would work for people without problems.
Yes, that is true. However it also speaks to the extremely small usage of rawhide and the relative lack of variety in both test hardware and setups. I have no magical solutions to those problems. We simply need more people testing rawhide kernels in general.
josh
On 06/13/2012 01:39 PM, Josh Boyer wrote:
Yes, that is true. However it also speaks to the extremely small usage of rawhide and the relative lack of variety in both test hardware and setups. I have no magical solutions to those problems. We simply need more people testing rawhide kernels in general.
A lot of us from the QA community who are not on rawhide are the running the rawhide kernels on F17 ( I'm currently on 3.5.0-0.rc2.git0.1.fc18.x86_64 ) and I have not noticed ( hence not reported ) any issue.
I think the problem is actually the other way around those of us that are in the QA community aren't actually using official update GA kernel.
And getting reporters to actually stay on a GA release once rawhide gets in usable shape ( which by my experience is around alpha ) is a problem. ( one of the thing that proven tester concept was supposed to address to ensure we had people testing GA )
And to add insult to injury reporters on update-testing have been routinely in the past been redirected from the test list to the user list with any issue they have discovered.
Something I tried to address a while back amongst other things but certain people in our QA community want to keep the list rawhide only...
JBG
On Wed, Jun 13, 2012 at 9:14 AM, M A Young m.a.young@durham.ac.uk wrote:
On Wed, 13 Jun 2012, Nikola Pajkovsky wrote:
imo, kernel maintainers should have released 3.3.8 or 3.4.1, not 3.4.0 for f17
I believe the 3.4.0 kernel package was effectively 3.4.1 anyway.
Yes, it was. Justin added the patches queued for the 3.4.1 release before he submitted the build.
josh
On Wed, 2012-06-13 at 12:51 +0200, Roman Kennke wrote:
I cannot believe that I am the only one on an Intel Wifi chip.
I haven't yet read the rest of the thread, but I just wanted to point out this common fallacy. As a general rule, all $VENDOR_$DEVICETYPE devices do not act the same. You cannot assume that all Intel wifi adapters are subject to the same bug you saw. You can't even assume that the same adapter _in another system_ would be subject to the same bug. This goes for everything: you can't assume all Intel video adapters behave the same, or all AMD video adapters, or all Broadcom wireless adapters, or _anything_.
I wish it _were_ true, believe me. It'd make testing (and I'm sure development) a lot easier if all we had to do was have one of each type of device from each manufacturer and then we could happily say 'okay, this kernel works with every piece of hardware in existence'. Boy, that'd be nice. Sadly, it just isn't the case at all.
Practical extensions from this rule:
1. Just because your NVIDIA graphics card doesn't work, don't assume that the developers / testers are such idiots they didn't test (whatever it is) on _any_ NVIDIA graphics cards at all.
2. Just because your Intel bluetooth adapter doesn't work, don't go around telling people on forums, IRC etc that they can't use any Intel bluetooth adapter with Fedora, because it's perfectly likely that lots of others work just fine.
(Both of these are real-life examples I've seen more than once).
The only good grounds for acting as if you're subject to a bug which is affecting a very broad class of hardware is if you have detailed reports from multiple other people with devices in the same class of hardware, detailed enough that it is clear they are all suffering from the same bug. Your own personal experience is _never_ sufficient grounds for concluding that the bug you're experiencing affects a much broader class of devices.
On 2012/06/13 11:22 (GMT-0700) Adam Williamson composed:
Your own personal experience is _never_ sufficient grounds for concluding that the bug you're experiencing affects a much broader class of devices.
Is "never" appropriate even if one's own experience is with 3 systems? 7 systems? 13 systems? 40 systems? Never say never, or always. ;-)
On Wed, 2012-06-13 at 14:33 -0400, Felix Miata wrote:
On 2012/06/13 11:22 (GMT-0700) Adam Williamson composed:
Your own personal experience is _never_ sufficient grounds for concluding that the bug you're experiencing affects a much broader class of devices.
Is "never" appropriate even if one's own experience is with 3 systems? 7 systems? 13 systems? 40 systems? Never say never, or always. ;-)
*fist shake*
Felix Miata wrote:
Is "never" appropriate even if one's own experience is with 3 systems? 7 systems? 13 systems? 40 systems? Never say never, or always. ;-)
That can widen the class of affected devices, but from there to "all Intel WiFi" is still a long stretch. (For a starter, last I checked they didn't even all use the same driver, but some legacy chipsets had legacy drivers. Then AFAIK there are different code paths in iwlwifi for different device generations, and also different firmware. And then of course different hardware behaves differently. Plus, WiFi issues also tend to depend on the exact configuration you're using, in particular on how the access point is set up. And finally there can be interaction with other hardware or drivers on the machine.)
Kevin Kofler
On 06/14/2012 04:54 AM, Kevin Kofler wrote:
Felix Miata wrote:
Is "never" appropriate even if one's own experience is with 3 systems? 7 systems? 13 systems? 40 systems? Never say never, or always. ;-)
That can widen the class of affected devices, but from there to "all Intel WiFi" is still a long stretch. (For a starter, last I checked they didn't even all use the same driver, but some legacy chipsets had legacy drivers. Then AFAIK there are different code paths in iwlwifi for different device generations, and also different firmware. And then of course different hardware behaves differently. Plus, WiFi issues also tend to depend on the exact configuration you're using, in particular on how the access point is set up. And finally there can be interaction with other hardware or drivers on the machine.)
And do not forget the BIOS version, especially for suspend-related ACPI bugs.