Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
I'm working with the reporter right now to investigate and hopefully get this fixed, but in the meantime - and this is in fact our standard advice anyway, but it bears repeating - DON'T RUN 'dnf update' INSIDE A DESKTOP.
Running the update process inside a desktop just gives it all the more opportunity to crash somehow. If the terminal app crashes, the update crashes. If the desktop crashes, the update crashes.
I don't want to get in the KDE folks' bad graces, but this likely could also affect KDE's graphical update system, so I'd advise against using that for the present too.
If you're using Workstation, the offline update system is expressly designed to minimize the likelihood of this kind of problem, so please do consider using it. Otherwise, at least run 'dnf update' in a VT - hit ctrl-alt-f3 to get a VT console login prompt, log in, and do it there. Don't do it inside your desktop.
Thanks folks!
On Oct 4, 2016 8:52 AM, "Adam Williamson" adamwill@fedoraproject.org wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
I once hosed a RHEL5 system when an ssh terminal running yum died. Sigh.
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" <adamwill@fedoraproject.org mailto:adamwill@fedoraproject.org> wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
On Tue, Oct 4, 2016 at 9:09 AM, Stephen Gallagher sgallagh@redhat.com wrote:
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" <adamwill@fedoraproject.org mailto:adamwill@fedoraproject.org> wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
On Tue, 4 Oct 2016 09:51:16 -0700 Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
Much less so. In the server case you have usually ssh, bash and dnf, in the desktop case you have X, possibly wayland, tons of graphics libraries, the terminal application you are using and all it's libraries, and a shell and dnf. There's just a lot more there to possibly crash.
kevin
On 4 October 2016 at 13:05, Kevin Fenzi kevin@scrye.com wrote:
On Tue, 4 Oct 2016 09:51:16 -0700 Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
Much less so. In the server case you have usually ssh, bash and dnf, in the desktop case you have X, possibly wayland, tons of graphics libraries, the terminal application you are using and all it's libraries, and a shell and dnf. There's just a lot more there to possibly crash.
As a systems administrator, I am left with at least 2 different fail states:
1) I reboot the box and have no idea what is going on because it never came back. 2) I run dnf update and I know it got to sshd or glibc when my connection to the server went away.
In either case I am hosed and if I am running a cloud of servers.. hosed in scale (or hosed as a service?). The part we really want to do is try and make it so if we are hosed.. how do we get it to be less hosed? Because this is going to happen in any case in some form no matter what.. when it does how do we recover in scale?
On Tue, Oct 4, 2016 at 10:05 AM, Kevin Fenzi kevin@scrye.com wrote:
On Tue, 4 Oct 2016 09:51:16 -0700 Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
Much less so. In the server case you have usually ssh, bash and dnf, in the desktop case you have X, possibly wayland, tons of graphics libraries, the terminal application you are using and all it's libraries, and a shell and dnf. There's just a lot more there to possibly crash.
My point is that a lot of this exposure could be avoided. Sure, there's a decent chance that updating packages will crash running programs. But, unless one of those programs is dnf, rpm, or systemd, that shouldn't be an excuse to blow up the whole upgrade. I've had Firefox blow up many times due to concurrent dnf, but this doesn't hose my system. Having gnome-terminal or X or Wayland die shouldn't be any more dangerous.
kevin
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
So, yes, offline updates or ostree-style updates are better in many respects, but as long as we provide plain dnf, I think it would be worth the small amount of effort to make dnf robust against the terminal dying.
--Andy
On Tue, Oct 4, 2016 at 11:26 AM, Andrew Lutomirski luto@mit.edu wrote:
On Tue, Oct 4, 2016 at 10:05 AM, Kevin Fenzi kevin@scrye.com wrote:
On Tue, 4 Oct 2016 09:51:16 -0700 Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
Much less so. In the server case you have usually ssh, bash and dnf, in the desktop case you have X, possibly wayland, tons of graphics libraries, the terminal application you are using and all it's libraries, and a shell and dnf. There's just a lot more there to possibly crash.
My point is that a lot of this exposure could be avoided. Sure, there's a decent chance that updating packages will crash running programs. But, unless one of those programs is dnf, rpm, or systemd, that shouldn't be an excuse to blow up the whole upgrade.
I think it's avoided by propagating the point adamw started the thread for.
User: Doctor, it hurts when I do this! DocAdamW: So then don't do that!
Do users need an INFO message when running 'dnf update' to kill off the myth that without a warning it's expected to be reliable? Maybe.
I've had Firefox blow up many times due to concurrent dnf, but this doesn't hose my system. Having gnome-terminal or X or Wayland die shouldn't be any more dangerous.
I don't understand how you arrive at this conclusion. dnf is sitting on the top of a house of cards when it's running in Terminal. If anything below it dies, dnf dies and by extension so is rpm. Could dnf be put into it's own session or scope (whatever it's called), and would that prevent it from dying if the whole GUI died? Even if true, how does the user know to wait XX minutes before hitting the reset button? I think it's a rabbit hole, just stop touching the owie.
Chris Murphy writes:
I don't understand how you arrive at this conclusion. dnf is sitting on the top of a house of cards when it's running in Terminal. If anything below it dies, dnf dies and by extension so is rpm. Could dnf be put into it's own session or scope (whatever it's called), and
Has anyone ever heard of sigaction()? Just wondering.
would that prevent it from dying if the whole GUI died? Even if true, how does the user know to wait XX minutes before hitting the reset button?
I heard of a command called "ps" that might come in useful, in situations like that.
On Tue, Oct 4, 2016 at 10:37 AM, Chris Murphy lists@colorremedies.com wrote:
On Tue, Oct 4, 2016 at 11:26 AM, Andrew Lutomirski luto@mit.edu wrote:
On Tue, Oct 4, 2016 at 10:05 AM, Kevin Fenzi kevin@scrye.com wrote:
On Tue, 4 Oct 2016 09:51:16 -0700 Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
Much less so. In the server case you have usually ssh, bash and dnf, in the desktop case you have X, possibly wayland, tons of graphics libraries, the terminal application you are using and all it's libraries, and a shell and dnf. There's just a lot more there to possibly crash.
My point is that a lot of this exposure could be avoided. Sure, there's a decent chance that updating packages will crash running programs. But, unless one of those programs is dnf, rpm, or systemd, that shouldn't be an excuse to blow up the whole upgrade.
I think it's avoided by propagating the point adamw started the thread for.
User: Doctor, it hurts when I do this! DocAdamW: So then don't do that!
Do users need an INFO message when running 'dnf update' to kill off the myth that without a warning it's expected to be reliable? Maybe.
I've pondered this a bit, and I think I disagree with you. A more apt analogy would be:
User: Doctor, it hurts when I walk. Doctor: So don't walk.
The System Administrator's Guide tells people to use dnf [1]. I agree that, in the long run, online updates of the system are probably the wrong solution. But the long run isn't here, and we could do a whole lot better than we do right now with a straightforward change.
I've had Firefox blow up many times due to concurrent dnf, but this doesn't hose my system. Having gnome-terminal or X or Wayland die shouldn't be any more dangerous.
I don't understand how you arrive at this conclusion. dnf is sitting on the top of a house of cards when it's running in Terminal. If anything below it dies, dnf dies and by extension so is rpm. Could dnf be put into it's own session or scope (whatever it's called), and would that prevent it from dying if the whole GUI died?
Yes. dnf would prepare the transaction, solve dependencies, etc, and then kick off a service to do the actual work. The service would pipe the progress state and such back to the client.
Even if true, how does the user know to wait XX minutes before hitting the reset button? I think it's a rabbit hole, just stop touching the owie.
In my entire experience administering Linux machines, I have *never* had serious package problems due to a transaction failing for any reason *other* than loss of the session that the package manager is running in. So I really do think it's a 90% solution to the problem that doesn't even require changing anyone's workflow.
So I filed the RFE. I think this should also be a prerequisite for the KillUserProcesses change.
https://bugzilla.redhat.com/show_bug.cgi?id=1382224
[1] https://docs.fedoraproject.org/en-US/Fedora/24/html/System_Administrators_Gu...
On Wed, 2016-10-05 at 20:38 -0700, Andrew Lutomirski wrote:
I don't understand how you arrive at this conclusion. dnf is sitting
on the top of a house of cards when it's running in Terminal. If anything below it dies, dnf dies and by extension so is rpm. Could dnf be put into it's own session or scope (whatever it's called), and would that prevent it from dying if the whole GUI died?
Yes. dnf would prepare the transaction, solve dependencies, etc, and then kick off a service to do the actual work. The service would pipe the progress state and such back to the client.
FWIW, someone told me today (in rather colorful terms) that apt-get is designed exactly this way. It survives the death of its controlling terminal (and, allegedly - again, I don't know this from personal experience - has decent support for completing failed transactions if the apt-get process itself crashes).
On 10/06/2016 05:45 AM, Adam Williamson wrote:
On Wed, 2016-10-05 at 20:38 -0700, Andrew Lutomirski wrote:
I don't understand how you arrive at this conclusion. dnf is sitting
on the top of a house of cards when it's running in Terminal. If anything below it dies, dnf dies and by extension so is rpm. Could dnf be put into it's own session or scope (whatever it's called), and would that prevent it from dying if the whole GUI died?
Yes. dnf would prepare the transaction, solve dependencies, etc, and then kick off a service to do the actual work. The service would pipe the progress state and such back to the client.
FWIW, someone told me today (in rather colorful terms) that apt-get is designed exactly this way. It survives the death of its controlling terminal (and, allegedly - again, I don't know this from personal experience - has decent support for completing failed transactions if the apt-get process itself crashes).
apt-get and dpkg cannot survive arbitrary crashes, but in general, the combination will rerun the equivalent of %preinst and %postinst scripts, to make sure that all installed packages are in a properly configured state.
It also intercepts ^C, so that you don't end up with an inconsistent package database simply because the administrator hit ^C in the wrong terminal.
Florian
Andrew Lutomirski writes:
My point is that a lot of this exposure could be avoided. Sure, there's a decent chance that updating packages will crash running programs. But, unless one of those programs is dnf, rpm, or systemd, that shouldn't be an excuse to blow up the whole upgrade.
I agree.
As far as I'm concerned, the only possible valid reason for the system to end up in an inconsistent state would be if the whole thing got SIGKILLed.
And I strongly doubt that an X crash would SIGKILL some process started by dnf. SIGHUP would be more likely. Perhaps SIGTERM, but neither of those should result in dnf blowing chunks and messing things up. Either the most recent transaction should be rolled back, or completed anyway.
The notion that an X crash would result in such situation is quite … inexplicable. There's simply no valid, logical reason for that, and I'm quite disappointed to hear that.
Firefox blow up many times due to concurrent dnf, but this doesn't hose my system. Having gnome-terminal or X or Wayland die shouldn't be any more dangerous.
Agreed.
And even if dnf itlsef is SIGKILLed, that may certainly result in a system being temporarily in an inconsistent state, but I would expect that unless something like glibc was in the process of being unpacked, it should be recoverable.
On Tue, Oct 4, 2016 at 10:05 AM, Kevin Fenzi kevin@scrye.com wrote:
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
OK, I'm completely confused. First of all, I've never seen a message such as:
Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
and I have NEVER used the graphical update since the first release of Fedora. I've always used yum or dnf. As I mentioned earlier in the past I've found the graphical tools to be quirky at best - perhaps that has changed, but since the command line has always worked for me, I've stuck with it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
As far as rebooting after every update? Huh? Who does that? Are we Windows? We're suppose to be moving toward the time when you won't need to reboot for kernel updates. Perhaps this is only a GNOME problem... but when I used GNOME years ago, I never experienced it. I also have never seen it with KDE or LxQT.
On Tue, 2016-10-04 at 10:39 -0700, Gerald B. Cox wrote:
and I have NEVER used the graphical update since the first release of Fedora. I've always used yum or dnf. As I mentioned earlier in the past I've found the graphical tools to be quirky at best - perhaps that has changed, but since the command line has always worked for me, I've stuck with it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
It's always been dangerous. It works fine all the time until it doesn't, and then you're left with a pile of broken bits that you get to spend all afternoon fixing.
It's pretty simple, really: a process running in a terminal inside a graphical desktop will crash if the terminal app crashes, or if the desktop crashes, or if X crashes. If any of those things happens, your update process just dies instantly, leaving whatever bits it hadn't done yet...undone. This is a situation it's technically more or less impossible to *fully* recover from (it's more or less impossible to figure out and execute precisely whatever scriptlet actions should have happened but did not), and is a pain to more-or-less-practically recover from (you get to hack up some crappy script to detect duplicate packages, run rpm -e --justdb --noscripts on the old NEVR and run dnf reinstall on the new NEVR...)
I'm not entirely sure if the KDE graphical updater is safer or not, but I don't think it is, because I think it again effectively just runs the update transaction inside the KDE session, where it will die partway through if KDE or X crashes.
The current GNOME update workflow, however, is the most reliable we have, because it downloads the updates then boots to a minimal systemd target with as few things running as possible to install the updates, then boots back to the normal system. This is far, far safer than running the update inside a desktop.
On Tue, Oct 4, 2016 at 1:53 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2016-10-04 at 10:39 -0700, Gerald B. Cox wrote:
and I have NEVER used the graphical update since the first release of Fedora. I've always used yum or dnf. As I mentioned earlier in the past I've found the graphical tools to be quirky at best - perhaps that has changed, but since the command line has always worked for me, I've stuck with it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
It's always been dangerous. It works fine all the time until it doesn't, and then you're left with a pile of broken bits that you get to spend all afternoon fixing.
It's pretty simple, really: a process running in a terminal inside a graphical desktop will crash if the terminal app crashes, or if the desktop crashes, or if X crashes. If any of those things happens, your update process just dies instantly, leaving whatever bits it hadn't done yet...undone. This is a situation it's technically more or less impossible to *fully* recover from (it's more or less impossible to figure out and execute precisely whatever scriptlet actions should have happened but did not), and is a pain to more-or-less-practically recover from (you get to hack up some crappy script to detect duplicate packages, run rpm -e --justdb --noscripts on the old NEVR and run dnf reinstall on the new NEVR...)
I'm not entirely sure if the KDE graphical updater is safer or not, but I don't think it is, because I think it again effectively just runs the update transaction inside the KDE session, where it will die partway through if KDE or X crashes.
The current GNOME update workflow, however, is the most reliable we have, because it downloads the updates then boots to a minimal systemd target with as few things running as possible to install the updates, then boots back to the normal system. This is far, far safer than running the update inside a desktop.
I have never heard of anyone reaching out to the KDE PK frontend developers for supporting this mechanism. As I recall, it required special development to get working in GNOME Software. Heck, even the system upgrade stuff required custom development and special plugins for GNOME Software. If you're going to suggest that people do it this way, then perhaps you should be reaching out to the Plasma Discover and Apper developers to ensure that it works from there too...
On 10/04/2016 02:01 PM, Neal Gompa wrote:
On Tue, Oct 4, 2016 at 1:53 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2016-10-04 at 10:39 -0700, Gerald B. Cox wrote:
and I have NEVER used the graphical update since the first release of Fedora. I've always used yum or dnf. As I mentioned earlier in the past I've found the graphical tools to be quirky at best - perhaps that has changed, but since the command line has always worked for me, I've stuck with it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
It's always been dangerous. It works fine all the time until it doesn't, and then you're left with a pile of broken bits that you get to spend all afternoon fixing.
It's pretty simple, really: a process running in a terminal inside a graphical desktop will crash if the terminal app crashes, or if the desktop crashes, or if X crashes. If any of those things happens, your update process just dies instantly, leaving whatever bits it hadn't done yet...undone. This is a situation it's technically more or less impossible to *fully* recover from (it's more or less impossible to figure out and execute precisely whatever scriptlet actions should have happened but did not), and is a pain to more-or-less-practically recover from (you get to hack up some crappy script to detect duplicate packages, run rpm -e --justdb --noscripts on the old NEVR and run dnf reinstall on the new NEVR...)
I'm not entirely sure if the KDE graphical updater is safer or not, but I don't think it is, because I think it again effectively just runs the update transaction inside the KDE session, where it will die partway through if KDE or X crashes.
The current GNOME update workflow, however, is the most reliable we have, because it downloads the updates then boots to a minimal systemd target with as few things running as possible to install the updates, then boots back to the normal system. This is far, far safer than running the update inside a desktop.
I have never heard of anyone reaching out to the KDE PK frontend developers for supporting this mechanism. As I recall, it required special development to get working in GNOME Software. Heck, even the system upgrade stuff required custom development and special plugins for GNOME Software. If you're going to suggest that people do it this way, then perhaps you should be reaching out to the Plasma Discover and Apper developers to ensure that it works from there too...
The requisite functionality was built in PackageKit specifically to ensure that it was available for any desktop that wanted to use it. Obviously, since GNOME Software developers were the ones who developed it, they did the work to make sure it functions properly there.
Nothing stops KDE or any other mechanism from using this functionality (it's exposed in the public PackageKit API). See my other replies on this thread for information on how you could do it from the `pkcon` CLI in a pinch...
On Tue, 2016-10-04 at 14:01 -0400, Neal Gompa wrote:
I have never heard of anyone reaching out to the KDE PK frontend developers for supporting this mechanism. As I recall, it required special development to get working in GNOME Software. Heck, even the system upgrade stuff required custom development and special plugins for GNOME Software. If you're going to suggest that people do it this way, then perhaps you should be reaching out to the Plasma Discover and Apper developers to ensure that it works from there too...
I'm giving users advice on how not to break their systems. I'm not really in the business of telling desktops how to code their update mechanisms.
On Tue, Oct 4, 2016 at 10:53 AM, Adam Williamson <adamwill@fedoraproject.org
wrote:
The current GNOME update workflow, however, is the most reliable we have, because it downloads the updates then boots to a minimal systemd target with as few things running as possible to install the updates, then boots back to the normal system. This is far, far safer than running the update inside a desktop.
Interesting... since I don't use GNOME that is probably why I wasn't aware of this.
I found this: https://lists.fedoraproject.org/pipermail/users/2015-March/459603.html
From what I've read so far, KDE does not do "offline" updates.... I'm still researching so maybe that has changed, but in the above mentioned thread, KDE did not support it.
I understand the theoretical exposure, but I guess what I'm missing is how offline updates eliminates that risk? There is still a plethora of things which could interfere with a normal completion of the update process. Seems to me it would be more worthwhile to build in better error recovery within DNF than to always require "offline" - especially since the incidence of failure (at least anecdotally) just isn't that high. Instead of dealing with the problem (failed updates and error recovery) - this approach just tries to avoid it by always requiring a reboot. Kind of defeats the purpose of being able to update a kernel without a reboot, if your going to always reboot for other updates - and of course the majority of updates don't require a reboot.
IMHO the risk/benefit ratio is way off on this approach to the problem - but hey, that's just me - and I'm a KDE user who isn't using it.
On Tue, Oct 4, 2016 at 12:51 PM, Gerald B. Cox gbcox@bzb.us wrote:
I understand the theoretical exposure, but I guess what I'm missing is how offline updates eliminates that risk?
The system reboots to system-update.target which is a minimal environment. It's basically the kernel, systemd, rpm and maybe dnf. Then it reboots again, with graphical.target.
There is still a plethora of things which could interfere with a normal completion of the update process.
No.
Seems to me it would be more worthwhile to build in better error recovery within DNF than to always require "offline" - especially since the incidence of failure (at least anecdotally) just isn't that high.
Sufficiently impractical that it's not possible. This is why offline updates exists. It's why work is being done on ostree>rpm-ostree>atomic host, which affects the entire build system, deployments, updates, and eventually all of the mirrors. It's why Microsoft and Apple don't allow anything other than offline updates. It's why openSUSE has spent a ton of resources, and a few bloody noses, getting completely atomic updates working with Btrfs and snapper, with very fine rollback capabilities. There's a reason why so many different experts at system updates have looked at this problem and just say, yeah no, not anymore of that.
Instead of dealing with the problem (failed updates and error recovery) - this approach just tries to avoid it by always requiring a reboot.
Yes. But it's also considered a stop gap. It's not the permanent state. A better way forward is in development.
Kind of defeats the purpose of being able to update a kernel without a reboot, if your going to always reboot for other updates - and of course the majority of updates don't require a reboot.
Fedora doesn't enable or use live patching for the kernel.
IMHO the risk/benefit ratio is way off on this approach to the problem - but hey, that's just me - and I'm a KDE user who isn't using it.
I think your risk assessment is deficient and unconvincing. This has been explained in great detail by those working on the various solutions to the problem. Read those, then provide contra arguments if you want. Otherwise this is sortof a noisy conversation.
On Tue, Oct 4, 2016 at 12:22 PM, Chris Murphy lists@colorremedies.com wrote:
IMHO the risk/benefit ratio is way off on this approach to the problem -
but
hey, that's just me - and I'm a KDE user who isn't using it.
I think your risk assessment is deficient and unconvincing. This has been explained in great detail by those working on the various solutions to the problem. Read those, then provide contra arguments if you want. Otherwise this is sortof a noisy conversation.
Well, it could be that this is just an issue in the GNOME environment. I can't really comment on that because I haven't used it for years. Don't get upset just because I don't agree with you.. You haven't produced any hard facts.
On Tue, Oct 4, 2016 at 1:37 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 4, 2016 at 12:22 PM, Chris Murphy lists@colorremedies.com wrote:
IMHO the risk/benefit ratio is way off on this approach to the problem - but hey, that's just me - and I'm a KDE user who isn't using it.
I think your risk assessment is deficient and unconvincing. This has been explained in great detail by those working on the various solutions to the problem. Read those, then provide contra arguments if you want. Otherwise this is sortof a noisy conversation.
Well, it could be that this is just an issue in the GNOME environment. I can't really comment on that because I haven't used it for years.
It is not a GNOME specific environment, this has already been explained.
Don't get upset just because I don't agree with you.. You haven't produced any hard facts.
You can't make me angry. Meanwhile, you're constantly citing only your own anecdotal evidence and then extrapolating beyond it to a much broader population without any hard facts.
Like I said, others have been working on this for a while, and they have documented it and they've included their hard facts. You just haven't bothered to make yourself aware of anything beyond your own experience.
On Tue, Oct 4, 2016 at 12:52 PM, Chris Murphy lists@colorremedies.com wrote:
Like I said, others have been working on this for a while, and they have documented it and they've included their hard facts. You just haven't bothered to make yourself aware of anything beyond your own experience.
Interesting... I posted a few links - still reading... haven't found anything that changes my opinion. If you have a link with some facts and statistics on failure please post.
On Tue, Oct 4, 2016 at 1:57 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 4, 2016 at 12:52 PM, Chris Murphy lists@colorremedies.com wrote:
Like I said, others have been working on this for a while, and they have documented it and they've included their hard facts. You just haven't bothered to make yourself aware of anything beyond your own experience.
Interesting... I posted a few links - still reading... haven't found anything that changes my opinion. If you have a link with some facts and statistics on failure please post.
Let me Google that for you? No thanks. I told you the various projects that are working on safer system updates, they've written tons about this on their lists. That's where I learned about this stuff.
On Tue, Oct 4, 2016 at 1:02 PM, Chris Murphy lists@colorremedies.com wrote:
On Tue, Oct 4, 2016 at 1:57 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 4, 2016 at 12:52 PM, Chris Murphy lists@colorremedies.com wrote:
Like I said, others have been working on this for a while, and they have documented it and they've included their hard facts. You just haven't bothered to make yourself aware of anything beyond your own experience.
Interesting... I posted a few links - still reading... haven't found anything that changes my opinion. If you have a link with some facts and statistics on failure please post.
Let me Google that for you? No thanks. I told you the various projects that are working on safer system updates, they've written tons about this on their lists. That's where I learned about this stuff.
Chris, I have been googling... and reading quite a bit... what I said was that I have yet to find what you are claiming.
Chris Murphy writes:
Seems to me it would be more worthwhile to build in better error recovery within DNF than to always require "offline" - especially since the incidence of failure (at least anecdotally) just isn't that high.
Sufficiently impractical that it's not possible.
Wrong.
I can make any process survive an X shutdown, using an amazing tool called "tmux".
Why can't dnf do the same?
It's why openSUSE has spent a ton of resources, and a few bloody noses, getting completely atomic updates working with Btrfs and snapper, with very fine rollback capabilities.
You do not need atomic updates to install a signal handler for SIGHUP or SIGTERM. And maybe issue a setsid() call, beforehand.
This shouldn't be rocket science.
Once upon a time, Sam Varshavchik mrsam@courier-mta.com said:
I can make any process survive an X shutdown, using an amazing tool called "tmux".
Why can't dnf do the same?
Because dnf would have to reimplement tmux's (or screen's) TTY handling, which is far outside the scope of a package manager. If you want to run dnf under tmux/screen, then run it under tmux/screen.
You do not need atomic updates to install a signal handler for SIGHUP or SIGTERM. And maybe issue a setsid() call, beforehand.
That's not enough. dnf prints running information to standard out/error, which are connected to the terminal via a TTY. As soon as the terminal process is killed, the TTY is closed, and the next write by dnf will cause an error.
Now, the error could be ignored, but it is hard to say there's a legitimate path to follow after such an error. A system-level program like dnf should generally not ignore such errors. Any RPM script that writes to standard out/error (they're not supposed to do that, but some do, sometimes inadvertently) will also fail.
If you want dnf to survive the terminal exiting, run it under something that handles that, like tmux or screen. Heck, alias dnf to "screen dnf" (or the tmux equivalent) if you like. Don't try to make dnf reimplement all that functionality.
Chris Adams writes:
Once upon a time, Sam Varshavchik mrsam@courier-mta.com said:
I can make any process survive an X shutdown, using an amazing tool called "tmux".
Why can't dnf do the same?
Because dnf would have to reimplement tmux's (or screen's) TTY handling, which is far outside the scope of a package manager. If you want to run dnf under tmux/screen, then run it under tmux/screen.
By my rough estimate, the relevant bits of "TTY handling" that are needed to survive the terminal going away comprise no more than four or five system calls.
You do not need atomic updates to install a signal handler for SIGHUP or SIGTERM. And maybe issue a setsid() call, beforehand.
That's not enough. dnf prints running information to standard out/error, which are connected to the terminal via a TTY. As soon as the terminal process is killed, the TTY is closed, and the next write by dnf will cause an error.
Mea culpa. I forgot about SIGPIPE, my fault. Ok, let's just add SIGPIPE to the list of blocked signals.
I'm up to six system calls, now.
Now, the error could be ignored, but it is hard to say there's a legitimate path to follow after such an error. A system-level program like dnf should generally not ignore such errors. Any RPM script that writes to standard out/error (they're not supposed to do that, but some do, sometimes inadvertently) will also fail.
Splendid. Have the scripts' stdout redirected to a pipe, that dnf will read, and attempt to log it to the terminal on its own, and deal with the fallout.
If you want dnf to survive the terminal exiting, run it under something that handles that, like tmux or screen. Heck, alias dnf to "screen dnf" (or the tmux equivalent) if you like. Don't try to make dnf reimplement all that functionality.
Okee-dokee. That's a reasonable proposition. But only if "reasonable" is defined in a fairly limited way; namely "work-around for lack of resilience and fault tolerance in a criticial system update tool".
On Tue, Oct 4, 2016 at 5:49 PM, Sam Varshavchik mrsam@courier-mta.com wrote:
Chris Murphy writes:
Seems to me it would be more worthwhile to build in better error recovery within DNF than to always require "offline" - especially since the incidence of failure (at least anecdotally) just isn't that high.
Sufficiently impractical that it's not possible.
Wrong.
I can make any process survive an X shutdown, using an amazing tool called "tmux".
Why can't dnf do the same?
I suggested earlier in the thread that it automatically put itself in its own scope. But I don't know if that solves this problem, and even if it does it's only one part of a much bigger set of problems that can cause updates to implode. In this case, sure maybe dnf survives, but X is gone, the user has no idea what just happened, they have no idea dnf is still applying their updates in the background, so they don't know they shouldn't reboot.
When you start making the list of things that have to work exactly right for system updates in a GUI , you get basically two things:
1. fuck it, we're making the user log out and do the update offline, and;
2. fuck it, we're doing something completely different ala OSTree, ala Snapper, so the user can do a rollback if things go wrong.
The first is what happens on the other 90% of the world's computers, including Android system updates. Consider that. The second exists, right now, and it works and it's a lot better than what the rest of the world uses.
Strictly speaking Fedora doesn't make you do the first one, but it's *well* understood for a long time how fragile this is which is why offline updates was created.
It's why openSUSE has spent a ton of resources, and a few bloody noses, getting completely atomic updates working with Btrfs and snapper, with very fine rollback capabilities.
You do not need atomic updates to install a signal handler for SIGHUP or SIGTERM. And maybe issue a setsid() call, beforehand.
This shouldn't be rocket science.
OK you clearly don't understand the complexity. So go ahead and do things your way, and when you don't like the result from hitting the Hurt Me Button, complain and criticize people all you want, wait for the silence from those who could not give two shits, which means: stop doing things wrong, start doing them correctly, or fix it all yourself Mr. Genius. And then go write your own updater.
OK Everyone.. enough. If you want to keep at it, take it off-list.
Chris Murphy writes:
I suggested earlier in the thread that it automatically put itself in its own scope. But I don't know if that solves this problem, and even if it does it's only one part of a much bigger set of problems that can cause updates to implode. In this case, sure maybe dnf survives, but X is gone, the user has no idea what just happened, they have no idea dnf is still applying their updates in the background, so they don't know they shouldn't reboot.
Well, perhaps I misunderstood the situation at hand. But I'm under the impression that, right now, dnf just blows up and leaves things in an inconsistent state.
Maybe this is too much of a radical idea, but it makes sense to me to address this first, and then maybe worry about what is the best way for dnf to forge ahead when it realizes that its terminal is gone.
Right now, if I'm in a middle of a dnf update, and I try to start another dnf operation, the second operation will fail. So dnf is smart enough to exclusive-lock itself. I also have a dim recollection of some plugin that said something about blocking system shutdowns, in some context. I'm a bit hazy on that, but in any case, this is not something that must be solved on day 1. A non-sophisticated user is not going to drop to a terminal prompt, and execute dnf manually. I think that it's a reasonable proposition that someone who does do that, and ends up killing X for some reason, will be able to figure out if dnf is still running, or not, in the background.
When you start making the list of things that have to work exactly right for system updates in a GUI , you get basically two things:
fuck it, we're making the user log out and do the update offline, and;
fuck it, we're doing something completely different ala OSTree, ala
Snapper, so the user can do a rollback if things go wrong.
The first is what happens on the other 90% of the world's computers, including Android system updates.
But ordinary regular app updates will happily run on cruise control, without bringing the system down into single user mode. If Android can do that, I see no reason why Fedora can't, either. The only time you need to reboot an Android device is for a kernel-level update.
Strictly speaking Fedora doesn't make you do the first one, but it's *well* understood for a long time how fragile this is which is why offline updates was created.
Well, this is a surprise to me. I guess my faith in dnf was misplaced.
I spent my time in the trenches. I spent a lot of time writing system daemons that I expected to recover from SIGKILL automatically. It wasn't exactly easy, but it was not an impossible task either. I'm not expecting any medals for that, just mentioning the fact that it's not, and should not be, considered to be a lost art. As I'm told it is now, apparently.
It's why openSUSE has spent a ton of resources, and a few bloody noses, getting completely atomic updates working with Btrfs and snapper, with very fine rollback capabilities.
You do not need atomic updates to install a signal handler for SIGHUP or SIGTERM. And maybe issue a setsid() call, beforehand.
This shouldn't be rocket science.
OK you clearly don't understand the complexity. So go ahead and do
Yes, I do not. But I am more than willing to be educated. Please explain to me the complexity in `setsid()`, and a few calls to `sigaction()`, and maybe an occasional `fork()`, here and there. Granted, when bleep hits the fan, you wouldn't immediately know where things stand without further digging. But, at least you are not going to end up with an unbootable brick, as long as you don't panic, and take things one step at a time.
things your way, and when you don't like the result from hitting the Hurt Me Button, complain and criticize people all you want, wait for the silence from those who could not give two shits, which means: stop doing things wrong, start doing them correctly, or fix it all yourself Mr. Genius. And then go write your own updater.
You know, actually I did, about ten years ago, I think – don't recall the exact timeframe. Nothing earth-shattering; just the same basic functionality as rpm: basic install/update/delete, and some dependency tracking.
And, by the way, an atomic package commit. If the process was SIGKILL-ed in a middle of a transaction – which might've involved a multi-package update – the install would finish automatically, when it got restarted. I spent some time on that because I recall that at the time rpm would do that too. I am certain that at least some point when you started rpm after something blew up, it would yell at you that a previous transaction hasn't been finished, and offered to clean it up for you.
So, it seems that this doesn't happen anymore. I guess that's the price of progress.
On Tue, 2016-10-04 at 20:43 -0400, Sam Varshavchik wrote:
But ordinary regular app updates will happily run on cruise control, without bringing the system down into single user mode. If Android can do that, I see no reason why Fedora can't, either. The only time you need to reboot an Android device is for a kernel-level update.
No, in fact, it's for any *system level* update. Any change to the underlying system (as opposed to an app) requires the full reboot treatment. Only updates to app packages don't.
The reason Android can do fairly good app updates is precisely because it does exactly what Flatpak and Snappy are trying to do for Linux: hard separation between app space and system space. Flatpak and Snappy didn't just spring fully formed from a vacuum, they're very obviously the product of someone using Android and/or iOS and going 'huh, maybe we should do that'.
We can't realistically do it with the 'distribution is just a big pile of RPMs' model. GNOME folks thought we could, for a while, then realized they were wrong.
On 10/05/2016 03:00 AM, Adam Williamson wrote:
On Tue, 2016-10-04 at 20:43 -0400, Sam Varshavchik wrote:
But ordinary regular app updates will happily run on cruise control, without bringing the system down into single user mode. If Android can do that, I see no reason why Fedora can't, either. The only time you need to reboot an Android device is for a kernel-level update.
No, in fact, it's for any *system level* update. Any change to the underlying system (as opposed to an app) requires the full reboot treatment. Only updates to app packages don't.
Even that depends on what the “app” is doing. If the format of data files changes in an incompatible way (or just their location on the file system
The reason Android can do fairly good app updates is precisely because it does exactly what Flatpak and Snappy are trying to do for Linux: hard separation between app space and system space.
I seriously doubt that is the reason.
This looks much more relevant:
https://developer.android.com/guide/components/activities.html#ImplementingLifecycleCallbacks
Basically, the system can kill applications which do not run in the foreground at *any* time to recover resources. This facility can also be used to transparently update system components on which applications depend.
We can't realistically do it with the 'distribution is just a big pile of RPMs' model.
The lack of application state management which mirrors what mobile platforms are doing is hardly an RPM issue.
Florian
On Tue, Oct 4, 2016 at 5:43 PM, Sam Varshavchik mrsam@courier-mta.com wrote:
Strictly speaking Fedora doesn't make you do the first one, but it's
*well* understood for a long time how fragile this is which is why offline updates was created.
Well, this is a surprise to me. I guess my faith in dnf was misplaced.
Here also. The DNF team has done an excellent job in obfuscating the horror IMO because (knock on wood) I've always been very impressed with the features and functionality. Seems a shame that all those nice bells and whistles will now be hidden behind some gui or pkcon.
I've been scrambling reading threads trying to understand what exactly is the exposure here. The only thing I could find that quantified the risk was in this kde thread:
*"...Updating online works 99.8% of the time. The 0.1% time it will corrupt random bits of your file-system, and 0.1% of the time it will leave you vulnerable to the security issue you thought you just "fixed". The only way to fix this so that online updates are safe is to redesign the centralised shared package model we use for distributing applications. The workaround is to use offline updates..."*
On Tue, 2016-10-04 at 18:59 -0700, Gerald B. Cox wrote:
I've been scrambling reading threads trying to understand what exactly is the exposure here. The only thing I could find that quantified the risk was in this kde thread:
You're never really going to be able to 'quantify the risk', because we don't have solid enough data. We don't know exactly how many millions of people are running Fedora in how many millions of configurations and just how many of them have ever had a dnf update failure. We don't even know any single one of those things. You're kind of on a hiding to nothing there. Point is, live update processes can and do go wrong. We certainly have enough records of that. Dig through bugzilla and you'll find plenty. I can think of ~4 cases that actually *did* get reported and precisely identified since F21. The more stuff running under the update process, the more likely problems are to happen.
All dnf's 'nice features' aren't really there for a system update, are they? You can use all of its nice features for doing other things. When you update the system, all you want is...an updated system. pkcon and GNOME's offline update flow achieve that just fine.
Adam Williamson writes:
All dnf's 'nice features' aren't really there for a system update, are they?
Well what are they there for, then?
You can use all of its nice features for doing other things.
Like what? I thought that the only thing dnf does is update the system. Oh, yeah, sure you can query metadata on available packages too.
When
you update the system, all you want is...an updated system. pkcon and GNOME's offline update flow achieve that just fine.
Hmmm… Right now, running "pkcon get-updates" claims "There are no updates available at this time.", meanwhile "dnf upgrade" shows 68 packages that can be updated. "pkcon refresh" doesn't change anything.
On Tue, 2016-10-04 at 22:36 -0400, Sam Varshavchik wrote:
Adam Williamson writes:
All dnf's 'nice features' aren't really there for a system update, are they?
Well what are they there for, then?
Querying the package database and installing new packages. A system update, on the other hand, is...it's not *configurable*, really. You say 'update my system, please!' and you get an updated system. That's kind of it. I'm just struggling to think which of the 'features and functionality' of dnf Gerald was talking about in the context of doing a system update. I think the most I'd ever use would be --enablerepo / --disablerepo , and --nogpgcheck (and hopefully we'll never need that one again anyway, now).
But I was talking to Gerald, not you. I'm not really sure why you keep jumping in and prolonging this thread - you apparently didn't hit the bug, and you certainly know enough about the tech to drive your own personal updates however you choose. What exactly is your goal here?
Hmmm… Right now, running "pkcon get-updates" claims "There are no updates available at this time.", meanwhile "dnf upgrade" shows 68 packages that can be updated. "pkcon refresh" doesn't change anything.
I don't actually have much experience of driving it through pkcon, I've only ever used the GNOME Software route, where the refresh button more or less seems to do what it claims. I don't use the offline updates stuff on my own servers, I'm okay with just letting dnf-automatic do them. My personal fuzzy evaluation is that the you get about 90% of the safety gain going from 'run in a terminal in X' to 'run in a VT', offline updates get you another 10%, and I'm okay with that tradeoff for my machines because they're just not terribly important. If I was running a gajillion dollar server farm I'd probably have different considerations.
On Tue, Oct 4, 2016 at 8:30 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Querying the package database and installing new packages. A system update, on the other hand, is...it's not *configurable*, really. You say 'update my system, please!' and you get an updated system. That's kind of it. I'm just struggling to think which of the 'features and functionality' of dnf Gerald was talking about in the context of doing a system update.
Adam, one of my favorite commands (which has kept me out of trouble) is "dnf history". Chris posted some excellent links so I'm going to say more there.
On Wed, 2016-10-05 at 06:18 -0700, Gerald B. Cox wrote:
On Tue, Oct 4, 2016 at 8:30 PM, Adam Williamson adamwill@fedoraproject.org wrote:
Querying the package database and installing new packages. A system update, on the other hand, is...it's not *configurable*, really. You say 'update my system, please!' and you get an updated system. That's kind of it. I'm just struggling to think which of the 'features and functionality' of dnf Gerald was talking about in the context of doing a system update.
Adam, one of my favorite commands (which has kept me out of trouble) is "dnf history". Chris posted some excellent links so I'm going to say more there.
Ah, that is a point: I *really really hate* that transaction history is not shared between dnf and PackageKit, so 'dnf history' doesn't show transactions that were run by pkcon or GNOME Software. That drives me clean up the damn wall.
On 5/10/2016 12:36, Sam Varshavchik wrote:
Adam Williamson writes:
All dnf's 'nice features' aren't really there for a system update, are they?
Well what are they there for, then?
You can use all of its nice features for doing other things.
Like what? I thought that the only thing dnf does is update the system. Oh, yeah, sure you can query metadata on available packages too.
I think there is a misunderstanding here on the use of "system".
I believe Adam is saying that upgrading packages for the underlying OS is where the problem is, but you can upgrade packages in other layers of the OS fine.
e.g. upgrading glibc can be an issue, upgrading vi should be fine.
Upgrading emacs is undefined.
Cheers, Jeff.
On Wed, 2016-10-05 at 13:34 +1000, Jeff Fearn wrote:
On 5/10/2016 12:36, Sam Varshavchik wrote:
Adam Williamson writes:
All dnf's 'nice features' aren't really there for a system update, are they?
Well what are they there for, then?
You can use all of its nice features for doing other things.
Like what? I thought that the only thing dnf does is update the system. Oh, yeah, sure you can query metadata on available packages too.
I think there is a misunderstanding here on the use of "system".
I believe Adam is saying that upgrading packages for the underlying OS is where the problem is, but you can upgrade packages in other layers of the OS fine.
e.g. upgrading glibc can be an issue, upgrading vi should be fine.
Upgrading emacs is undefined.
No, that's not what I meant. I was replying to Gerald's post (though I missed quoting the relevant bit, which led to some confusion):
"The DNF team has done an excellent job in obfuscating the horror IMO because (knock on wood) I've always been very impressed with the features and functionality. Seems a shame that all those nice bells and whistles will now be hidden behind some gui or pkcon."
I was just confused as to what 'features and functionality' would be 'hidden' by a GUI or pkcon in the case of doing a system update, since I can't really think what 'features and functionality' you really use when running dnf update.
On 10/04/2016 08:12 PM, Chris Murphy wrote:
The first is what happens on the other 90% of the world's computers, including Android system updates. Consider that. The second exists, right now, and it works and it's a lot better than what the rest of the world uses.
What do you mean by 'system'? Android apps update online just fine. Fedora does not have this distinction betwen apps and core system: it's just a bunch of packages. I think it is a mistake to require reboot for _every_ update, even though, as you say, even user apps sometimes _cannot_ be updated online (*). I am comfortable with online updates, and I take more drastic action---restart apps and/or reboot---only when I actually see problems.
(*) because of running processes, or library/API dependencies, or changes in communication/bus protocols.
On Wed, Oct 05, 2016 at 05:14:11PM -0400, Przemek Klosowski wrote:
What do you mean by 'system'? Android apps update online just fine. Fedora does not have this distinction betwen apps and core system: it's just a bunch of packages. I think it is a mistake to require reboot for _every_ update, even though, as you say, even user apps sometimes _cannot_ be updated online (*). I am comfortable with online updates, and I take more drastic action---restart apps and/or reboot---only when I actually see problems.
This is one of the reasons we _might_ want to consider delivering most user apps as Flatpaks (and server apps as containers).
On Wed, Oct 5, 2016 at 3:14 PM, Przemek Klosowski przemek.klosowski@nist.gov wrote:
On 10/04/2016 08:12 PM, Chris Murphy wrote:
The first is what happens on the other 90% of the world's computers, including Android system updates. Consider that. The second exists, right now, and it works and it's a lot better than what the rest of the world uses.
What do you mean by 'system'? Android apps update online just fine.
It depends on what's being updated. If Google Play is updated, I've experienced erratic and sluggish applications until the phone is rebooted. System is anything that's a package rather than the blobs they stick into various partitions and mandate a reboot to apply.
I think it is a mistake to require reboot for _every_ update, even though, as you say, even user apps sometimes _cannot_ be updated online (*). I am comfortable with online updates, and I take more drastic action---restart apps and/or reboot---only when I actually see problems.
If you read all the posts I cited, it's pretty clear that the notion updates are really being used if you haven't rebooted is entirely illusory/wishful thinking. There's a good chance you wouldn't know there were a problem.The only way to be sure is to reboot.
On 10/5/2016 2:40 PM, Chris Murphy wrote:
On Wed, Oct 5, 2016 at 3:14 PM, Przemek Klosowski przemek.klosowski@nist.gov wrote:
I think it is a mistake to require reboot for _every_ update, even though, as you say, even user apps sometimes _cannot_ be updated online (*). I am comfortable with online updates, and I take more drastic action---restart apps and/or reboot---only when I actually see problems.
If you read all the posts I cited, it's pretty clear that the notion updates are really being used if you haven't rebooted is entirely illusory/wishful thinking. There's a good chance you wouldn't know there were a problem.The only way to be sure is to reboot.
I don't know what the dnf equivalent is, but isn't that precisely what the 'needs-restarting' command provided?
And speaking more broadly, a competent RH-derived-distro administrator is already aware of this concern about libraries not taking effect. They also have the tools (lsof, etc.) at their disposal to make their own decisions on a live system about whether a service (or the host itself, for things like glibc) should be bounced when updating a library package.
It feels like any auto-restart behavior (outside of the %post scriptlets in the service packages themselves) should be advisory in nature.
-jc
On Wed, Oct 05, 2016 at 03:26:10PM -0700, Japheth Cleaver wrote:
I don't know what the dnf equivalent is, but isn't that precisely what the 'needs-restarting' command provided?
DNF has the tracer plugin for doing exactly that, including printing a list of commands to restart the affected services, if possible, or advising a manual restart of specific programs, the login session or the whole system otherwise, depending on the package set.
On Tue, 2016-10-04 at 11:51 -0700, Gerald B. Cox wrote:
I understand the theoretical exposure, but I guess what I'm missing is how offline updates eliminates that risk? There is still a plethora of things which could interfere with a normal completion of the update process.
It doesn't eliminate it, it minimizes it. It makes the 'plethora' much smaller. Basically, I think, unless pid 1 crashes or dnf itself crashes, the update is going to complete.
Seems to me it would be more worthwhile to build in better error recovery within DNF
I'm not sure that's a path that'd really get anywhere terribly profitable. You could probably make some kinds of improvements to it, I guess, but I'm not sure it's ever going to be possible to say 'meh, no big deal if the updater crashes halfway through the update'.
than to always require "offline" - especially since the incidence of failure (at least anecdotally) just isn't that high.
The point is that it doesn't *have* to be high for it to be a problem. Having this happen one time is one time too many.
(But FWIW, it seems like in this case the crash is pretty much reliably reproducible on at least one affected system, and would happen *any* time systemd-udev is updated.)
Instead of dealing with the problem (failed updates and error recovery) - this approach just tries to avoid it by always requiring a reboot. Kind of defeats the purpose of being able to update a kernel without a reboot, if your going to always reboot for other updates - and of course the majority of updates don't require a reboot.
I think you're kind of conflating two things here. The offline update thing isn't about 'make sure there's a reboot so the updates are fully applied'. The first reboot is to get to the clean minimal environment where the update can be safely run, the second reboot is to get back *out* of that environment. It's not really 'avoid[ing the problem] by always requiring a reboot', it's avoiding the problem by running the update in a minimal environment, which happens to *imply* a couple of reboots (though we've had discussions about ways it could be done with just one reboot).
Some people do seem to place great stock in the 'update without rebooting' thing, but it's fundamentally not entirely safe with RPM updates. Even the offline update approach isn't 100% safe, though it's a lot closer than updating in X and a little closer than updating from a VT. Fedora, AFAIK, doesn't have any kind of focus on the whole 'update the kernel without rebooting' thing, so I'm not sure why you're bringing that up.
If anything, the potential future Fedora is working towards is one where the system is deployed and updated via ostree and apps are deployed and updated via flatpak or DOCKAH DOCKAH DOCKAH. That's a design where it's feasible to talk about safe updates without reboots.
IMHO the risk/benefit ratio is way off on this approach to the problem - but hey, that's just me - and I'm a KDE user who isn't using it.
If you have an affected system, then the risk of running an update that includes systemd-udev from inside X is 100%. That's a pretty high ratio.
Adam Williamson writes:
It's pretty simple, really: a process running in a terminal inside a graphical desktop will crash if the terminal app crashes, or if the desktop crashes, or if X crashes.
It should take me about five minutes to write a process that will continue happily along if its terminal, desktop, or X crashes. Especially if the process doesn't need to talk to X in the first place. Like dnf.
I'm quite astonished to read this. Something is wrong, here. Either I have suffered a stroke, today, or I seriously misunderstood some POSIX fundamentals, for the last twenty years.
For chrissakes, there's a bleeping program in Fedora called "tmux" that has managed to accomplished the herculean feat of surviving X shutting down.
I just performed a simple experiment.
1. Opened a terminal window
2. Executed tmux
3. In a tmux session executed "cat >/dev/null"
4. Logged out from the desktop, and only because CTRL-ALT-BACKSPACE has been disabled "for our convenience", but to the tmux sesssion running in a terminal this is indistinguishable from losing the X session unexpectedly.
5. Guess what? After logging back in, opening a terminal window, and executed "tmux attach", I found my "cat" program happily waiting to dump more stuff into /dev/null.
Really, the silly excuse that, somehow, X shutting down is a valid reason to dnf to blow chunks is embarassing. This whole thread is mind-boggling.
I'll be more than happy to eat crow, if proven wrong by pointing out some detail that I missed; but if I consider anything short of a direct SIGKILL to a dnf process (or the entire system crashing due to power loss, etc) ending up with an inconsistent state, to be a BUG.
No, really, I challenge anyone to reproduce this amazing feat of tmux surviving an unexpected X shutdown and then tell me why dnf cannot do the same.
On Tue, 2016-10-04 at 19:47 -0400, Sam Varshavchik wrote:
I'm quite astonished to read this. Something is wrong, here. Either I have suffered a stroke, today, or I seriously misunderstood some POSIX fundamentals, for the last twenty years.
Oh fer Pete's sake, would you please ratchet down the rhetoric and stop posting the same thing over and over again? You've made your position clear and now you're just polluting the thread. I'm trying to advise real users on how to ensure they don't suffer real problems, not start an epic argument about how to write update tools. I'm sure the dnf developers are happy to take submissions through the usual channels.
On Tue, Oct 4, 2016 at 11:39 AM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 4, 2016 at 10:05 AM, Kevin Fenzi kevin@scrye.com wrote:
Well, the problem there, what do you mean by 'support'?
In this case lots of people use dnf for updates, so IMHO it would be "we will try and keep this working, and fix anything we can, but do understand that there's a low level problem here that something could mess up updates in progress, if you want to be more sure of not hitting problems, use the offline updates in your graphical desktop"
OK, I'm completely confused. First of all, I've never seen a message such as:
Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
That was a suggested change to inform the user.
It's an OK idea. With "atomic" deployments (those that are rpm-ostree based) the 'dnf update' command hard fails. You're expected to use 'rpm-ostree upgrade' which itself would be integrated into the graphical updater.
and I have NEVER used the graphical update since the first release of Fedora. I've always used yum or dnf. As I mentioned earlier in the past I've found the graphical tools to be quirky at best - perhaps that has changed, but since the command line has always worked for me, I've stuck with it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
This is old news. It's come up on this list numerous times. It's the *entire* reason why offline updates exists. Not doing them offline is know to entail all kinds of non-deterministic risks.
As far as rebooting after every update? Huh? Who does that?
Strictly speaking it's not necessary for every update, there's just no mechanism for knowing for sure what updates entail more risk than others. You'll notice that once an application is installed, whether by dnf or Gnome Software, it's considered part of the system. There's no separation of OS upgrades from application updates. The former entail a lot more risk than the latter. In the near future, I expect flatpak applications can be updated in place, once complete you'd get a prompt to relaunch the application to use the update one, or something like that.
Are we Windows?
Yes, we're Windows. Is that what you want to hear? I don't understand why you think this is a question to be taken seriously rather than ridiculous.
And by the way, macOS does the same thing. The main difference there is application updates rarely require reboots, because applications are self contained. You do have to quit them for the update to get applied though.
On 10/04/2016 01:19 PM, Chris Murphy wrote:
Strictly speaking it's not necessary for every update, there's just no mechanism for knowing for sure what updates entail more risk than others. You'll notice that once an application is installed, whether by dnf or Gnome Software, it's considered part of the system. There's no separation of OS upgrades from application updates. The former entail a lot more risk than the latter. In the near future, I expect flatpak applications can be updated in place, once complete you'd get a prompt to relaunch the application to use the update one, or something like that.
dnf install python3-dnf-plugins-extras-tracer
On Tue, Oct 4, 2016 at 12:49 PM, Michael Cronenworth mike@cchtml.com wrote:
On 10/04/2016 01:19 PM, Chris Murphy wrote:
Strictly speaking it's not necessary for every update, there's just no mechanism for knowing for sure what updates entail more risk than others. You'll notice that once an application is installed, whether by dnf or Gnome Software, it's considered part of the system. There's no separation of OS upgrades from application updates. The former entail a lot more risk than the latter. In the near future, I expect flatpak applications can be updated in place, once complete you'd get a prompt to relaunch the application to use the update one, or something like that.
dnf install python3-dnf-plugins-extras-tracer
That's cool but it's not the default behavior, and it also doesn't come in advance as part of all the information what will be updated. It'd be useful to know what needs restarting, including if a reboot is needed.
Looks like this might also be useful, but isn't included by default. http://dnf-plugins-core.readthedocs.io/en/latest/needs_restarting.html
On Tue, Oct 4, 2016 at 11:19 AM, Chris Murphy lists@colorremedies.com wrote:
Are we Windows?
Yes, we're Windows. Is that what you want to hear? I don't understand why you think this is a question to be taken seriously rather than ridiculous.
No, it was a rhetorical question. See my other response. Basically, IMHO this "solution" fails a simple risk/benefit analysis. Instead of dealing with the root issue which is error recovery it attempts to circumvent it at the expense of system availability.
On 4 October 2016 at 15:00, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 4, 2016 at 11:19 AM, Chris Murphy lists@colorremedies.com wrote:
Are we Windows?
Yes, we're Windows. Is that what you want to hear? I don't understand why you think this is a question to be taken seriously rather than ridiculous.
No, it was a rhetorical question. See my other response. Basically, IMHO this "solution" fails a simple risk/benefit analysis. Instead of dealing with the root issue which is error recovery it attempts to circumvent it at the expense of system availability.
This problem is not easily solvable by dnf or yum or anything else. You are standing on the branch of a tree. When you do an update you are cutting off branches of a tree and growing new ones. Sometimes you are cutting the trunk of the tree. The problem is that if you are cutting your own branch or the tree trunk.. you are sunk.
The problem is that modern software has a lot of branches. The tree looks like this https://twitter.com/NaturelsWeird/status/783006609765269504
Maybe that branch you cut doesn't look like it has anything to do with the one you are standing on.. but maybe it did and now you are hosed.
This has been a problem for as long as Red Hat Linux existed with a remote shell and people would end up with crashed systems because sshd died when an update was done. There were all kinds of 'hacks' to make it work mostly but there were still conditions where you could crash out your system. It also showed up every now and then on X updates which could also be 'hacked' around but was still racy. As the tree gets more complicated.. the more the hacks are moving from an 95% solution to a 0.95 * 0.95 * 0.95 ... and ending up with it being a 60% solution and just getting worse over time.
devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
Gerald B. Cox writes:
it. Apparently I've missed something along the way because now people are implying that using the command line tools from within GNOME or KDE are dangerous. What exactly is going on?
Somewhere along the way, it seems that quite a few sharks have been jumped over.
Sad.
Kevin, now you mention Wayland... Could it be possible to be a Wayland related issue? I have Nvidia but no Wayland and never an issue updating from desktop.
Cheers,
Sylvia
On Tue, Oct 4, 2016 at 10:51 AM, Andrew Lutomirski luto@mit.edu wrote:
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
Well it's not always risky, it depends on what's being updated. If it's applications, it's pretty safe unless they're running. There is a qualitative difference between server and workstation, so the warning may be misleading if it exists on server also.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
Yeah this is really debatable for best practices to expect users to know these things. You can argue if they're using dnf they should know better, and just accept that things can blow up, or help track them down so if possible this stuff can get fixed.
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
PackageKit is installed on server, so while I haven't tested it, it seems plausible it could be used for system updates. But I think the risk here is much less because dnf is a bit more isolated by not running in a GUI Terminal in a user session. But even here, a reboot is expected for many things, there's just no guarantee of state otherwise.
I think the thing that's unique with current offline updates compared to other platforms I use is the double reboot. The reboot to get to offline updates target where the update happens, and then a reboot to get back to graphical target. That does seem suboptimal to me, where other platforms just log out the user session, and drop to their reduced function "updates" state to do the update, and then there's one reboot.
Anyway, my opinion is that Workstation folks should use Gnome Software to do their updates, and if there's something wonky there, we need to get it fixed. That's the default and primary update method. Within something like 5 minutes of first boot after an installation, PackageKit is already downloading updated packages. Unless that's disabled, a 'dnf update' is going to unnecessarily download those packages in duplicate.
On 10/04/2016 12:51 PM, Andrew Lutomirski wrote:
On Tue, Oct 4, 2016 at 9:09 AM, Stephen Gallagher sgallagh@redhat.com wrote:
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" <adamwill@fedoraproject.org mailto:adamwill@fedoraproject.org> wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
``` sudo pkcon refresh force && \ sudo pkcon update --only-download && \ sudo pkcon offline-trigger ** sudo systemctl reboot ```
On 10/04/2016 01:20 PM, Stephen Gallagher wrote:
On 10/04/2016 12:51 PM, Andrew Lutomirski wrote:
On Tue, Oct 4, 2016 at 9:09 AM, Stephen Gallagher sgallagh@redhat.com wrote:
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" <adamwill@fedoraproject.org mailto:adamwill@fedoraproject.org> wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
By that standard, why do we support dnf at all?
$ sudo dnf upgrade Error: dnf upgrade is dangerous. Use PackageKit instead and reboot when asked.
I, for one, *like* not rebooting, and I'm perfectly capable of rebooting manually if stuff breaks. As far as I know, Fedora considers plain ol' dnf to be supported.
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
sudo pkcon refresh force && \ sudo pkcon update --only-download && \ sudo pkcon offline-trigger ** sudo systemctl reboot
That "**" should have been "&& "
Stephen Gallagher wrote:
On 10/04/2016 01:20 PM, Stephen Gallagher wrote:
On 10/04/2016 12:51 PM, Andrew Lutomirski wrote:
For server use, I'm not convinced that the offline update mechanism is supported (at the very least, I have no idea how to trigger it), and servers have the same issue.
sudo pkcon refresh force && \ sudo pkcon update --only-download && \ sudo pkcon offline-trigger ** sudo systemctl reboot
That "**" should have been "&& "
If there isn't a single command to do all of that, that can compete with "yum update" in simplicity, then that shows that offline updating isn't being seriously promoted for servers.
Björn Persson
I never needed to reboot. I just keep working on my stuff, when I'm done I turn the laptop off. Is there any reason to reboot right after updating?
Cheers,
Sylvia
I never needed to reboot. I just keep working on my stuff, when I'm done I turn the laptop off. Is there any reason to reboot right after updating?
That's actually a very common misunderstanding. People think that "yum/dnf update" leaves their system in a new updated stage. But it doesn't (completely). It never has. Only after a reboot are all your patches applied and active. Existing/running processes are rarely if ever reloaded. So when you update libraries, kernels etc. your system will keep running with the old versions of those libraries loaded. Remember, in Linux a inode is never deleted until the very last process has released it. So any read file handles will keep a file, even one you cannot see with ls, available.
The only real complete update you can do is one that does a full reboot. We do have a few tricks with DNF which will attempt to let you know what needs restarting. But you'll find that a good part of our updates requires a restart of most if not all your system, in order for the updates to become fully active.
This problem often shows itself on long running servers by a system not coming back up/online after a reboot. And nobody understand why - but it's pretty simple. Nobody tested that things worked after an update, and in most cases that requires a reboot. If you kernel, glibc or network control system gets updated, you'll need to reboot to reload them. Or of course take your network offline with everything running on your box (good luck!).
So it may look like that "it works just fine" but it's a deception. It's still running the older versions of libraries etc for most processes. If you start looking at what's actually running after your update, you'll find that a good part of your updated packages aren't running (yet) - and old versions are still active. If you're attempting to apply security updates, this is an important distinction and vital to be compliant. For us ordinary users, it's mostly an annoyance as features we wanted, aren't there right after an update, or programs starts failing after an update, as they attempt to dynamicly load in modules and find them "changed", at times not even compatible with the running older version, and things end up "strange". I've seen fonts go nuts, backgrounds disappear, general themes not working etc. after an update on a workstation, simply because the structure got changed - CSSes looking for files that no longer are there, etc.
// Peter
On Wed, Oct 05, 2016 at 01:57:20PM -0000, Peter Larsen wrote:
I never needed to reboot. I just keep working on my stuff, when I'm done I turn the laptop off. Is there any reason to reboot right after updating?
That's actually a very common misunderstanding. People think that "yum/dnf update" leaves their system in a new updated stage. But it doesn't (completely). It never has. Only after a reboot are all your patches applied and active. Existing/running processes are rarely if ever reloaded. So when you update libraries, kernels etc. your system will keep running with the old versions of those libraries loaded. Remember, in Linux a inode is never deleted until the very last process has released it. So any read file handles will keep a file, even one you cannot see with ls, available.
The only real complete update you can do is one that does a full reboot. We do have a few tricks with DNF which will attempt to let you know what needs restarting. But you'll find that a good part of our updates requires a restart of most if not all your system, in order for the updates to become fully active.
What you write is mostly true, at least for "user" programs. Various system services actually get restarted, e.g. all those that are running as systemd units and are have %systemd_postun_with_restart in %post. For servers that should be a majority of daemons (but not all, dbus-daemon being a notable exception). Systemd also supports seamless restart of socket-activated services, so it's even possible to restart daemons that are in live use without anybody noticing.
For users programs we have nothing like that unfortunately.
But yeah, I'm nitpicking here, the general rule is what you say: one has to restart the machine to make sure everything has bee restarted.
Zbyszek
This problem often shows itself on long running servers by a system not coming back up/online after a reboot. And nobody understand why - but it's pretty simple. Nobody tested that things worked after an update, and in most cases that requires a reboot. If you kernel, glibc or network control system gets updated, you'll need to reboot to reload them. Or of course take your network offline with everything running on your box (good luck!).
So it may look like that "it works just fine" but it's a deception. It's still running the older versions of libraries etc for most processes. If you start looking at what's actually running after your update, you'll find that a good part of your updated packages aren't running (yet) - and old versions are still active. If you're attempting to apply security updates, this is an important distinction and vital to be compliant. For us ordinary users, it's mostly an annoyance as features we wanted, aren't there right after an update, or programs starts failing after an update, as they attempt to dynamicly load in modules and find them "changed", at times not even compatible with the running older version, and things end up "strange". I've seen fonts go nuts, backgrounds disappear, general themes not working etc. after an update on a workstation, simply because the structure got changed - CSSes looking for files that no longer are there, etc.
On Tue, Oct 4, 2016 at 10:09 AM, Stephen Gallagher sgallagh@redhat.com wrote:
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" <adamwill@fedoraproject.org mailto:adamwill@fedoraproject.org> wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
It's not really workable without an atomic and out of tree update method, otherwise libraries are still yanked out from under running processes at some point. I've done this with nspawn (and chroot), taking snapshots of root, then applying the update to the snapshot, changing the bootloader to boot the updated snapshot. This is tedious but it's reliable in that pretty much anything bad can happen and it's only the fs tree being updated that can be broken. And only one reboot is needed.
The long term solution is rpm-ostree based Workstation where the currently running fs tree isn't touched either.
Chris Murphy wrote:
It's not really workable without an atomic and out of tree update method, otherwise libraries are still yanked out from under running processes at some point.
Running programs and their loaded libraries count as open files. Unixy filesystems don't delete open files. They change the directory entry to point to the new file and keep the old file around until it's closed. Therefore libraries are *not* yanked out from under running processes.
What tends to break is badly designed GUI programs that apparently don't keep their files open but still expect different files to be from the exact same version. Firefox and Seamonkey seem to do something like that. They tend to break in funny ways when updated. Many other GUI programs have no problems with getting updated while running.
Björn Persson
Stephen Gallagher wrote:
That seems like a waste of effort, considering we have the offline updates process which just boots into a special, minimalist environment with almost nothing but the updater running.
But the offline process is highly impractical, it requires you to interrupt all you were doing on every single update, just for the unlikely event that doing otherwise would crash X11. Making it so that the X11 crash will not crash the update process is, sure, slightly less effective (doesn't prevent potential data loss in the unlikely case X11 crashes), but also much more practical, while still providing the same reliability guarantees for the update process itself. So I don't see this being a waste of effort at all.
Kevin Kofler
On Tue, Oct 4, 2016 at 12:06 PM, Andrew Lutomirski luto@mit.edu wrote:
On Oct 4, 2016 8:52 AM, "Adam Williamson" adamwill@fedoraproject.org wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
I once hosed a RHEL5 system when an ssh terminal running yum died. Sigh.
This is pretty much what dnfdaemon does. However, dnfdaemon is only used by yumex-dnf[1] and dnfdragora[2].
To the best of my knowledge, there are no CLI clients for it.
[1]: https://github.com/timlau/yumex-dnf [2]: https://github.com/anaselli/dnfdragora
On 10/04/2016 12:06 PM, Andrew Lutomirski wrote:
How hard would it be to make dnf do the rpm transaction inside a proper system-level service (transient or otherwise)? This would greatly increase robustness against desktop crashes, ssh connection loss, KillUserProcs, and other damaging goofs.
I once hosed a RHEL5 system when an ssh terminal running yum died. Sigh.
Now that systemd kills processes on logout, and session timeout is required by security policies in many workplaces including mine, I fear we'll have more of those. I certainly had update sessions that involved hundreds of packages and took longer than the session timeout, whcih is often on ther order of several minutes.
On Wed, Oct 05, 2016 at 04:59:57PM -0400, Przemek Klosowski wrote:
Now that systemd kills processes on logout, and session timeout is required by security policies in many workplaces including mine, I fear we'll have more of those. I certainly had update sessions that involved hundreds of packages and took longer than the session timeout, whcih is often on ther order of several minutes.
Please remember that we have deferred enabling the kill-processes-on-logout setting until at least F26.
How is your session timeout implemented? If it's $TMOUT in bash, that shouldn't affect the _running_ command. Does it let you leave long-running processes with screen or tmux?
On 10/05/2016 05:24 PM, Matthew Miller wrote:
On Wed, Oct 05, 2016 at 04:59:57PM -0400, Przemek Klosowski wrote:
Now that systemd kills processes on logout, and session timeout is required by security policies in many workplaces including mine, I fear we'll have more of those. I certainly had update sessions that involved hundreds of packages and took longer than the session timeout, whcih is often on ther order of several minutes.
Please remember that we have deferred enabling the kill-processes-on-logout setting until at least F26.
How is your session timeout implemented? If it's $TMOUT in bash, that shouldn't affect the _running_ command. Does it let you leave long-running processes with screen or tmux?
I believe it's a setting in sshd. I will have to learn to run many things in tmux/screen: certainly dnf, but also all the science codes that we normally do. Actually, this will build our character: currently we kind-of depend on getting lucky if the command happens to run in background or knows to put itself in background on receivng a HUP.
On Tue, 04 Oct 2016 08:51:07 -0700 Adam Williamson adamwill@fedoraproject.org wrote:
I'm working with the reporter right now to investigate and hopefully get this fixed, but in the meantime - and this is in fact our standard advice anyway, but it bears repeating - DON'T RUN 'dnf update' INSIDE A DESKTOP.
I think I can confirm this advice. I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue. I don't use the graphical update tools, and always have PackageKit disabled to prevent duplicate downloads of updates.
On 10/04/2016 08:10 PM, stan wrote:
I think I can confirm this advice. I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue.
The problem with this is that the VT doesn't have a long history, so if you can't really see what dnf has done. (e.g. I visually search any rpmsave rpmorig rpmnew warning)
On Fri, 2016-10-07 at 21:11 +0200, Roberto Ragusa wrote:
On 10/04/2016 08:10 PM, stan wrote:
I think I can confirm this advice. I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue.
The problem with this is that the VT doesn't have a long history, so if you can't really see what dnf has done. (e.g. I visually search any rpmsave rpmorig rpmnew warning)
Several ways to handle this have already been discussed upthread, and another thing you can do is look at the 'dnf history info' output for the transaction; that contains all the same warnings that are shown real-time.
[...] I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue.
To the extent that the problem is that dnf gets interrupted when its xterm dies, can that be worked around by dnf SIG_IGN'ing SIGHUP / SIGPIPE? Users could still deliberately interrupt it with SIGINT, but a terminal closure could in theory let it finish unmolested.
- FChE
On Oct 7, 2016 1:29 PM, "Frank Ch. Eigler" fche@redhat.com wrote:
[...] I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue.
To the extent that the problem is that dnf gets interrupted when its xterm dies, can that be worked around by dnf SIG_IGN'ing SIGHUP / SIGPIPE? Users could still deliberately interrupt it with SIGINT, but a terminal closure could in theory let it finish unmolested.
Python is weird^Wspecial. See the bug I opened.
On Fri, Oct 7, 2016 at 4:33 PM, Andrew Lutomirski luto@mit.edu wrote:
On Oct 7, 2016 1:29 PM, "Frank Ch. Eigler" fche@redhat.com wrote:
[...] I always run dnf manually from the command line, in a VT logged in as root. And I can run X while doing this and I've never had a dnf update issue.
To the extent that the problem is that dnf gets interrupted when its xterm dies, can that be worked around by dnf SIG_IGN'ing SIGHUP / SIGPIPE? Users could still deliberately interrupt it with SIGINT, but a terminal closure could in theory let it finish unmolested.
Python is weird^Wspecial. See the bug I opened.
I must suggest that "nohup dnf whatever > logfile.out &" is your friend for exactly this sort of thing. stdin/stdout/stderr are all disconnected from the running session and a visible log is generated. And this is *much, much* easier than trying to repair legacy architectures of complexity to preserve the status of an associated terminal wound through X windows and Gnome and "sophisticated feature filled architecture of the week".
It doesn't fix the underlying mishandling of terminal session loss. But it can get you and others trapped through the short term.
On Sun, Oct 9, 2016 at 7:33 AM, Nico Kadel-Garcia nkadel@gmail.com wrote:
I must suggest that "nohup dnf whatever > logfile.out &" is your friend for exactly this sort of thing. stdin/stdout/stderr are all disconnected from the running session and a visible log is generated.
Thanks very much for taking the time to post... very simple and to the point.
Another method I've found for those concerned about using a terminal under a DE is to check out dnf-automatic: https://fedoraproject.org/wiki/AutoUpdates There is currently an open bug regarding the use of email notifications when using dnf-automatic: https://bugzilla.redhat.com/show_bug.cgi?id=1279773 but other than that, it appears to work fine.
For those wanting to know which processes need to be restarted after update, there is the appropriately named dnf needs-restarting: http://dnf-plugins-core.readthedocs.io/en/latest/needs_restarting.html
As the documentation mentions:
Note that in most cases a process should survive update of its binary and libraries it is using without requiring to be restarted for proper operation. There are however specific cases when this does not apply.
On Sun, Oct 9, 2016 at 10:01 AM, Gerald B. Cox gbcox@bzb.us wrote:
For those wanting to know which processes need to be restarted after update, there is the appropriately named dnf needs-restarting: http://dnf-plugins-core.readthedocs.io/en/latest/needs_restarting.html
I also just found: Project Tracer: what you should restart after "dnf upgrade" You can read about it here, if you're interested: https://goo.gl/ZFS29E
Preview snippet: ...
Every time you update your system, your currently running applications should be restarted otherwise you will be still running those old binaries before the update.
You have 3 options how to resolve it:
1. You ignore it completely and you patiently wait until your system crash. 2. Do Offline Updates http://fedoraproject.org/wiki/Features/OfflineSystemUpdates. 3. After restart you carefully restart only those applications and daemons, which needs to be restarted.
I really dislike Offline Updates as I see no reason for going offline for dozens of minutes. And there is only few scripts which helps you when you want to follow option 3. Most famous is needs-restarting http://man7.org/linux/man-pages/man1/needs-restarting.1.html from yum-utils. But the output is very ... ehm ... unfriendly. And it is very slow.
...
On 10 October 2016 at 00:04, Gerald B. Cox gbcox@bzb.us wrote:
You have 3 options how to resolve it:
- You ignore it completely and you patiently wait until your system
crash. 2. Do Offline Updates http://fedoraproject.org/wiki/Features/OfflineSystemUpdates. 3. After restart you carefully restart only those applications and daemons, which needs to be restarted.
You have still 4th option:
- create from snapshots separated boot environment (BE) - do upgrade in BE - restart system from new BE
To be honest only tis method solves all hazards which are not listed in your 3 points and additionally you will have possibility to create packages without any post install/uninstall scripts which are biggest risk factor of ANY upgrades. Really Linux needs to start learning from what has been done on Solaris more than decade ago. What old SySV packages maintainers found that majority of different customers problems on applying upgrades where generated by fully customizable post install/uninstall scripts. This was main driving force on implementing in IPS something which is called package action which is well tested sequence of operation in some class of installed resources.
BE has very precise properties. It keeps only one kernel instance in each BE and boot loader must collect on boot stage informations about available BEs to present them as boot menu. Whatever needs to be customized as command line boot parameter needs to be part of BE description. Only with above approach is possible to stop updating boot entries and stop fighting with many other problems with which all Linux distros are fighting more than one and half decade.
Again: to solve all upgrade issues it needs to be synchronized few technologies related to boot manager, package manager and used file systems. At least using btrfs, grub2 and rpm with removed post install/uninstall custom scripts is possible to solve finally all issues. It is possible to prepare packages in some way allowing to use post install/uninstall in case not using btrfs but it could be exact amin decision to accept some well known risk coming with choosing old/legacy approach.
Whatever would be not done if not done using above idea will be still with non-zero risk of well known and new issues. Some people must understand that whatever they will be trying to archive without radical change which Solaris acceped and implemented decade ago it will be only WASTE OF TIME.
kloczek
On Tue, Oct 11, 2016 at 2:38 PM, Tomasz Kłoczko kloczko.tomasz@gmail.com wrote:
You have still 4th option:
- create from snapshots separated boot environment (BE)
- do upgrade in BE
- restart system from new BE
To be honest only tis method solves all hazards which are not listed in your 3 points and additionally you will have possibility to create packages without any post install/uninstall scripts which are biggest risk factor of ANY upgrades.
Thats fine, but first of all - those aren't my points - they are from the link I included regarding Project Tracer. My comment was directed at folks who were concerned about running dnf from a terminal within a DE - and were interested in some type of risk mitigation now. Your suggestion requires a bit more work. As far as BTRFS is concerned however, I believe that ship has sailed. I used it for 4 years, but after the recent news regarding RAID I switched everything to XFS.
On Tue, Oct 11, 2016 at 4:17 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 2:38 PM, Tomasz Kłoczko kloczko.tomasz@gmail.com wrote:
You have still 4th option:
- create from snapshots separated boot environment (BE)
- do upgrade in BE
- restart system from new BE
To be honest only tis method solves all hazards which are not listed in your 3 points and additionally you will have possibility to create packages without any post install/uninstall scripts which are biggest risk factor of ANY upgrades.
Thats fine, but first of all - those aren't my points - they are from the link I included regarding Project Tracer. My comment was directed at folks who were concerned about running dnf from a terminal within a DE - and were interested in some type of risk mitigation now. Your suggestion requires a bit more work.
Running an update in the DE on an out of tree snapshot of the file system is fairly trivial. The harder part is merging the changes that happen from snapshot time to reboot time, and managing rollbacks.
As far as BTRFS is concerned however, I believe that ship has sailed. I used it for 4 years, but after the recent news regarding RAID
The only news about Btrfs RAID I can think of that you're referring to is the raid5 scrub bug that corrupts parity in a very specific situation. It's a bad bug, but it's also really rare. First it requires a data strip to contain corruption. During scrub, the data block fails checksum, Btrfs does a good reconstruction from parity and repairs the bad data strip, but then goes on to *sometimes* wrongly recomputing parity and overwriting the good parity with bad parity. So in effect, it has silently transposed the corruption from a data strip to parity strip. Normal operation, you get your data, uncorrupted. If you lose a drive, and now that bad parity is needed, it results in a bad reconstruction of data, which results in EIO because the bad data fails checksum. So to get this form of data loss you need an already bad data strip, a scrub that hits this particular bug, and then lose a device. But you don't get corrupt data propagated upward.
It's uncertain how this manifests on raid6, I haven't tested it. My expectation is the failed csum from reconstruction using p-parity will result in another attempt using q-parity, and then fixing up the data and p-parities if the reconstruction passes the data checksum.
Understand that in the identical situation with mdraid and lvm raid, a scrub check would only report a mismatch, it wouldn't tell you which was correct. And if you did a scrub repair, it will assume data strips are valid, and in this case it'd *always* result in the wrong parity being written over good.
Anyway it's a bad bug. But it's not correct to assign this bug to the other three raid profiles or Btrfs in general - where it has numerous features not all of which have the same maturity.
https://btrfs.wiki.kernel.org/index.php/Status
I switched everything to XFS.
There are many good and valid reasons to use it.
On Tue, Oct 11, 2016 at 4:48 PM, Chris Murphy lists@colorremedies.com wrote:
As far as BTRFS
is concerned however, I believe that ship has sailed. I used it for 4
years, but after the recent news regarding RAID
The only news about Btrfs RAID I can think of that you're referring to is the raid5 scrub bug that corrupts parity in a very specific situation. It's a bad bug, but it's also really rare.
That may be, but all the articles I read suggested "be afraid, be very afraid". In addition, https://goo.gl/V2IyFq basically just comes out and says, not to use it.
I knew that it was experimental when I first set it up years ago, but I never imagined it would still be experimental in 2016. I just got tired of waiting, and the statement that it would all most likely have to be rewritten was just the last straw for me. The only reason I was using it was because of the ease and flexibility to run Raid5/6. With that apparently nowhere now on the horizon, time to cut my loses and move on.
Anyway it's a bad bug. But it's not correct to assign this bug to the other three raid profiles or Btrfs in general - where it has numerous features not all of which have the same maturity.
Agreed, but in my mind the last episode throws some serious shade on the entire project. Once upon a time, there was talk about making the default in Fedora, but now - not so much. In the meantime XFS is being improved and bcachefs is being developed. Tick tock...
On Tue, Oct 11, 2016 at 6:33 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 4:48 PM, Chris Murphy lists@colorremedies.com wrote:
As far as BTRFS
is concerned however, I believe that ship has sailed. I used it for 4 years, but after the recent news regarding RAID
The only news about Btrfs RAID I can think of that you're referring to is the raid5 scrub bug that corrupts parity in a very specific situation. It's a bad bug, but it's also really rare.
That may be, but all the articles I read suggested "be afraid, be very afraid". In addition, https://goo.gl/V2IyFq basically just comes out and says, not to use it.
That's stale and overstated but I'm fine with dire warnings because otherwise people use experimental stuff and flip out when it face plants. What's needed are people who can tediously gather an autopsy report so it can be made better. And that's pretty much what's happening.
I knew that it was experimental when I first set it up years ago, but I never imagined it would still be experimental in 2016. I just got tired of waiting, and the statement that it would all most likely have to be rewritten was just the last straw for me. The only reason I was using it was because of the ease and flexibility to run Raid5/6. With that apparently nowhere now on the horizon, time to cut my loses and move on.
Uhh OK, well it's an entirely new implementation of parity raid, unlike md or ZFS ZRAID. It was first merged a bit over three years ago, and the scrub and device replacement code only came last year. And even still there's no concept of faulty devices - so you don't get behaviors like devices being ejected from the array on write error or massive numbers of reads. It's entirely reasonable to want something more mature. If your use case requires bitrot and silent data corruption protection and parity raid, the single option is OpenZFS ZRAID.
Anyway it's a bad bug. But it's not correct to assign this bug to the other three raid profiles or Btrfs in general - where it has numerous features not all of which have the same maturity.
Agreed, but in my mind the last episode throws some serious shade on the entire project. Once upon a time, there was talk about making the default in Fedora, but now - not so much. In the meantime XFS is being improved and bcachefs is being developed. Tick tock...
About the Fedora default, this recently came up on desktop@ so I'll just refer to that: https://lists.fedoraproject.org/archives/list/desktop@lists.fedoraproject.or...
On Tue, Oct 11, 2016 at 8:39 PM, Chris Murphy lists@colorremedies.com wrote:
On Tue, Oct 11, 2016 at 6:33 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 4:48 PM, Chris Murphy lists@colorremedies.com wrote:
That may be, but all the articles I read suggested "be afraid, be very afraid". In addition, https://goo.gl/V2IyFq basically just comes out and says, not to use it.
That's stale and overstated but I'm fine with dire warnings because otherwise people use experimental stuff and flip out when it face plants. What's needed are people who can tediously gather an autopsy report so it can be made better. And that's pretty much what's happening.
Well that link is from their wiki. Then of course there is this thread
which kind of steps through it all: https://goo.gl/rIyJ0R One of my favorite quotes:
"Wow. So it sees the data strip corruption, uses good parity on disk to
fix it, writes the fix to disk, recomputes parity for some reason but does it wrongly, and then overwrites good parity with bad parity? That's fucked."
About the Fedora default, this recently came up on desktop@ so I'll just refer to that: https://lists.fedoraproject.org/archives/list/desktop@ lists.fedoraproject.org/message/T6FNLLPJ7LICKSVTTPS4KSIDHOKUUPNC/
Yeah, and there was also this thread: https://goo.gl/wWHLVA Like I said, I believe that ship has pretty much sailed...
On Tue, Oct 11, 2016 at 6:33 PM, Gerald B. Cox gbcox@bzb.us wrote:
I knew that it was experimental when I first set it up years ago, but I never imagined it would still be experimental in 2016. I just got tired of waiting, and the statement that it would all most likely have to be rewritten was just the last straw for me. The only reason I was using it was because of the ease and flexibility to run Raid5/6. With that apparently nowhere now on the horizon, time to cut my loses and move on.
About the rewrite comment: that did not come from a developer, and is definitely overstated. In any case, rewrites are not inherently bad news, there's a bunch of OpenZFS videos from last yearss summit in which the developers talk about various things being completely rewritten from scratch, some things more than twice. So kinda par for the course, and given enough time things get rewritten anyway. XFS has had substantial changes over its history including numerous on disk format changes even before it found its way onto Linux.
On Tue, Oct 11, 2016 at 8:52 PM, Chris Murphy lists@colorremedies.com wrote:
About the rewrite comment: that did not come from a developer, and is definitely overstated. In any case, rewrites are not inherently bad news, there's a bunch of OpenZFS videos from last yearss summit in which the developers talk about various things being completely rewritten from scratch, some things more than twice. So kinda par for the course, and given enough time things get rewritten anyway. XFS has had substantial changes over its history including numerous on disk format changes even before it found its way onto Linux.
Could be, should be, may be... that's fine - but it all says the same thing... they don't know how much time it is going to take to fix - and who knows what their priority is to get around to it. The advantages over what already is available don't appear to be that compelling, especially when weighed with the risks.
When all this started I did some searches and found Kent Overstreet's page on bcachefs: https://goo.gl/U0UFfN
He had some words about the different filesystems - and had this to say about btrfs:
btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too quickly without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors who years ago hoped to one day roll out btrfs are now quietly migrating to xfs instead).
Then there is this quote: https://www.youtube.com/watch?v=79fvDDPaIoY&t=18m28s https://www.youtube.com/watch?v=79fvDDPaIoY&t=18m28s"Software that is designed/ intended to be reliable should not go through large periods of instability only to be written off as "prepubescence"."
On Wed, Oct 12, 2016 at 6:29 AM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 8:52 PM, Chris Murphy lists@colorremedies.com wrote:
About the rewrite comment: that did not come from a developer, and is definitely overstated. In any case, rewrites are not inherently bad news, there's a bunch of OpenZFS videos from last yearss summit in which the developers talk about various things being completely rewritten from scratch, some things more than twice. So kinda par for the course, and given enough time things get rewritten anyway. XFS has had substantial changes over its history including numerous on disk format changes even before it found its way onto Linux.
Could be, should be, may be... that's fine - but it all says the same thing... they don't know how much time it is going to take to fix - and who knows what their priority is to get around to it. The advantages over what already is available don't appear to be that compelling, especially when weighed with the risks.
So you are saying that you started using raid56 when it was brand new, before it had *any* kind of persistent repairing or device replacement and only now, due to a bug that manifests remarkably less bad than the normal behavior of everything else (minus ZFS), are you now complaining? So basically this is, "I want it now!" complaint. Because all the available information has been saying don't use raid56 for production, but you did so anyway. This is a subjective change in your evaluation. It has nothing to do with the state of Btrfs so you really shouldn't blame it when your requirements have clearly changed.
When all this started I did some searches and found Kent Overstreet's page on bcachefs: https://goo.gl/U0UFfN
He had some words about the different filesystems - and had this to say about btrfs:
btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too quickly without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors who years ago hoped to one day roll out btrfs are now quietly migrating to xfs instead).
I have heard from a couple developers that it was a victim of its own hype/success and too many feature additions without equivalent effort on error reporting, debugging, and fault injection tools. I have yet to hear a Btrfs developer say the core design or on disk format has anything to do with the problems, but to the contrary. The comment it's bigger than XFS is kinda funny, seeing as it does more than XFS, md, and LVM combined, so a proper comparison would be comparing all of them to Btrfs minus their user space tools (for sure Btrfs tools do not come anywhere near the metric ass tonne of switches or documentation in LVM or mdadm).
The Fedora report is simply nonsense. Fedora made no meaningful attempt to switch to Btrfs once, let alone multiple times. FESCo considered and approved it, with conditions attached. Josef kept pushing it off because he thought it wasn't ready. And then Josef moved on from Red Hat, and wasn't replaced. Characterizing this as "tried to switch" and "had to switch at the last minute" is at best hyperbole. It was a change proposal, and it never met the requirements of either the proposer or FESCo for it to proceed further. No changes in default happened in the installer that had to be reverted.
Perhaps the secret of fast and stable fs development is a single developer authored file system.
On Wed, Oct 12, 2016 at 3:29 PM, Chris Murphy lists@colorremedies.com wrote:
On Wed, Oct 12, 2016 at 6:29 AM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 8:52 PM, Chris Murphy lists@colorremedies.com wrote:
About the rewrite comment: that did not come from a developer, and is definitely overstated. In any case, rewrites are not inherently bad news, there's a bunch of OpenZFS videos from last yearss summit in which the developers talk about various things being completely rewritten from scratch, some things more than twice. So kinda par for the course, and given enough time things get rewritten anyway. XFS has had substantial changes over its history including numerous on disk format changes even before it found its way onto Linux.
Could be, should be, may be... that's fine - but it all says the same thing... they don't know how much time it is going to take to fix - and who knows what their priority is to get around to it. The advantages over what already is available don't appear to be that compelling, especially when weighed with the
risks.
So you are saying that you started using raid56 when it was brand new, before it had *any* kind of persistent repairing or device replacement and only now, due to a bug that manifests remarkably less bad than the normal behavior of everything else (minus ZFS), are you now complaining? So basically this is, "I want it now!" complaint. Because all the available information has been saying don't use raid56 for production, but you did so anyway. This is a subjective change in your evaluation. It has nothing to do with the state of Btrfs so you really shouldn't blame it when your requirements have clearly changed.
Well no, what I said was that I thought that they might have a somewhat stable product after three years... instead I get that a rewrite may be required with no idea of when it would ever be production ready. If you think that is acceptable, more power to you. I do not and I would venture to guess that many reasonable people would think three years would be ample time. My requirements haven't changed... I just think the time to fish or cut bait has come. I've cut bait.
When all this started I did some searches and found Kent Overstreet's
page
on
bcachefs: https://goo.gl/U0UFfN
He had some words about the different filesystems - and had this to say about btrfs:
btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too
quickly
without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors
who
years ago hoped to one day roll out btrfs are now quietly migrating to
xfs
instead).
I have heard from a couple developers that it was a victim of its own hype/success and too many feature additions without equivalent effort on error reporting, debugging, and fault injection tools. I have yet to hear a Btrfs developer say the core design or on disk format has anything to do with the problems, but to the contrary. The comment it's bigger than XFS is kinda funny, seeing as it does more than XFS, md, and LVM combined, so a proper comparison would be comparing all of them to Btrfs minus their user space tools (for sure Btrfs tools do not come anywhere near the metric ass tonne of switches or documentation in LVM or mdadm).
I can't speak for him, but don't believe that was his point. I read it as that the code was bloated.
The Fedora report is simply nonsense. Fedora made no meaningful attempt to switch to Btrfs once, let alone multiple times. FESCo considered and approved it, with conditions attached.
Well, we're getting into semantics here. I would call that a serious attempt - and there are many articles that are available that talk about Fedora discussing making BTRFS a default. If you don't consider any of those attempts "meaningful" I suppose that's a good thing.
Josef kept pushing it off because he thought it wasn't ready.
Smart man, he was right.
And then Josef moved on from Red Hat, and wasn't replaced. Characterizing this as "tried to switch" and "had to switch at the last minute" is at best hyperbole.
Ok, if you say so. Again, that wasn't the way the media reported it.
It was a change proposal, and it never met the requirements of either the proposer or FESCo for it to proceed further. No changes in default happened in the installer that had to be reverted.
So change proposals aren't meaningful attempts.... OK... good to know.
Perhaps the secret of fast and stable fs development is a single developer authored file system.
Not sure about that... but considering what has happened with BTRFS, that indeed could be the case. The facts certainly appear to support it.
On Wed, Oct 12, 2016 at 8:35 PM, Gerald B. Cox gbcox@bzb.us wrote:
On Wed, Oct 12, 2016 at 3:29 PM, Chris Murphy lists@colorremedies.com wrote:
On Wed, Oct 12, 2016 at 6:29 AM, Gerald B. Cox gbcox@bzb.us wrote:
On Tue, Oct 11, 2016 at 8:52 PM, Chris Murphy lists@colorremedies.com wrote:
About the rewrite comment: that did not come from a developer, and is definitely overstated. In any case, rewrites are not inherently bad news, there's a bunch of OpenZFS videos from last yearss summit in which the developers talk about various things being completely rewritten from scratch, some things more than twice. So kinda par for the course, and given enough time things get rewritten anyway. XFS has had substantial changes over its history including numerous on disk format changes even before it found its way onto Linux.
Could be, should be, may be... that's fine - but it all says the same thing... they don't know how much time it is going to take to fix - and who knows what their priority is to get around to it. The advantages over what already is available don't appear to be that compelling, especially when weighed with the risks.
So you are saying that you started using raid56 when it was brand new, before it had *any* kind of persistent repairing or device replacement and only now, due to a bug that manifests remarkably less bad than the normal behavior of everything else (minus ZFS), are you now complaining? So basically this is, "I want it now!" complaint. Because all the available information has been saying don't use raid56 for production, but you did so anyway. This is a subjective change in your evaluation. It has nothing to do with the state of Btrfs so you really shouldn't blame it when your requirements have clearly changed.
Well no, what I said was that I thought that they might have a somewhat stable product after three years... instead I get that a rewrite may be required with no idea of when it would ever be production ready. If you think that is acceptable, more power to you. I do not and I would venture to guess that many reasonable people would think three years would be ample time. My requirements haven't changed... I just think the time to fish or cut bait has come. I've cut bait.
When all this started I did some searches and found Kent Overstreet's page on
bcachefs: https://goo.gl/U0UFfN
He had some words about the different filesystems - and had this to say about btrfs:
btrfs, which was supposed to be Linux's next generation COW filesystem - Linux's answer to zfs. Unfortunately, too much code was written too quickly without focusing on getting the core design correct first, and now it has too many design mistakes baked into the on disk format and an enormous, messy codebase - bigger that xfs. It's taken far too long to stabilize as well - poisoning the well for future filesystems because too many people were burned on btrfs, repeatedly (e.g. Fedora's tried to switch to btrfs multiple times and had to switch at the last minute, and server vendors who years ago hoped to one day roll out btrfs are now quietly migrating to xfs instead).
I have heard from a couple developers that it was a victim of its own hype/success and too many feature additions without equivalent effort on error reporting, debugging, and fault injection tools. I have yet to hear a Btrfs developer say the core design or on disk format has anything to do with the problems, but to the contrary. The comment it's bigger than XFS is kinda funny, seeing as it does more than XFS, md, and LVM combined, so a proper comparison would be comparing all of them to Btrfs minus their user space tools (for sure Btrfs tools do not come anywhere near the metric ass tonne of switches or documentation in LVM or mdadm).
I can't speak for him, but don't believe that was his point. I read it as that the code was bloated.
The Fedora report is simply nonsense. Fedora made no meaningful attempt to switch to Btrfs once, let alone multiple times. FESCo considered and approved it, with conditions attached.
Well, we're getting into semantics here. I would call that a serious attempt - and there are many articles that are available that talk about Fedora discussing making BTRFS a default. If you don't consider any of those attempts "meaningful" I suppose that's a good thing.
Josef kept pushing it off because he thought it wasn't ready.
Smart man, he was right.
And then Josef moved on from Red Hat, and wasn't replaced. Characterizing this as "tried to switch" and "had to switch at the last minute" is at best hyperbole.
Ok, if you say so. Again, that wasn't the way the media reported it.
I'll say so. I was the only one ever pushing it, and only mildly at best. Each time it was a discussion about what would the requirements look like to switch over so I knew what targets needed to be hit. In the end I felt like it wasn't worth the effort to try and switch Fedora over, Suse has way more inhouse expertise and willingness to do the work so I figured that was good enough and just gave up trying to switch Fedora over.
It was a change proposal, and it never met the requirements of either the proposer or FESCo for it to proceed further. No changes in default happened in the installer that had to be reverted.
So change proposals aren't meaningful attempts.... OK... good to know.
They weren't meaningful attempts, like I said above they were attempts to get the conversation started and to see what work would be required. Every time I felt my time was way better spent working on btrfs than dealing with the Fedora community. Thanks,
Josef
Thanks for the warning. Fortunately I frequently run update with Gnome Software. Does that issue also affect "pkcon update" command as well?
On Tue, Oct 4, 2016 at 1:34 PM, Luya Tshimbalanga luya@fedoraproject.org wrote:
Thanks for the warning. Fortunately I frequently run update with Gnome Software. Does that issue also affect "pkcon update" command as well?
Likely. It's avoided if you use --only-download, enable the offline update trigger and reboot.
On Tue, Oct 4, 2016 at 1:34 PM, Luya Tshimbalanga <luya(a)fedoraproject.org> wrote:
Likely. It's avoided if you use --only-download, enable the offline update trigger and reboot.
Thanks for the tips.
Luya
On Tue, 2016-10-04 at 19:34 +0000, Luya Tshimbalanga wrote:
Thanks for the warning. Fortunately I frequently run update with Gnome Software. Does that issue also affect "pkcon update" command as well?
Yes, it would. The issue does have an element of hardware dependence also. What we've worked out so far (with thanks to Zbyszek) is basically this:
1. When you update systemd-udev , it restarts systemd-udev-trigger.service in %post 2. That calls `udevadm trigger --type=devices --action=add` 3. That seems to result in the video adapter effectively being replugged at the udev level
Now on *most* systems, this seems to just cause X to (metaphorically) blink, go "what the hell happened there?", shrug, and carry on. If your system is one of these, you won't really notice anything. On affected systems, however - which so far seems to be something like 'some systems with NVIDIA adapters, possibly ones with NVIDIA/Intel hybrid graphics' - it causes X to crash.
If you want to check if your system is affected you can just run 'systemctl restart systemd-udev-trigger.service' as root with a desktop running, and see if it crashes. Make sure you don't have anything running unsaved. :P
I confirm my system, an AMD/AMD hybrid graphics powered laptop, is affected by executing 'systemctl restart systemd-udev-trigger.service' as root on desktop. The desktop session crashed and the login screen comes afterwards. The same command on a dedicated NVIDIA graphics card is fairly harmless.
It seems the issue is affecting all hardware with hybrid graphics.
Luya
On Tue, 2016-10-04 at 20:34 +0000, Luya Tshimbalanga wrote:
I confirm my system, an AMD/AMD hybrid graphics powered laptop, is affected by executing 'systemctl restart systemd-udev- trigger.service' as root on desktop. The desktop session crashed and the login screen comes afterwards. The same command on a dedicated NVIDIA graphics card is fairly harmless.
It seems the issue is affecting all hardware with hybrid graphics.
That's great data, thanks - I'm gonna find a few other people to test, but it definitely sounds like 'hybrid graphics' is the trigger here.
On Tue, 2016-10-04 at 20:34 +0000, Luya Tshimbalanga wrote:
That's great data, thanks - I'm gonna find a few other people to test, but it definitely sounds like 'hybrid graphics' is the trigger here.
No problem. Glad the data helps.
Luya
On Tue, 2016-10-04 at 20:34 +0000, Luya Tshimbalanga wrote:
That's great data, thanks - I'm gonna find a few other people to test, but it definitely sounds like 'hybrid graphics' is the trigger here.
No problem. Glad the data helps.
Luya
On Tue, Oct 4, 2016 at 2:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2016-10-04 at 20:34 +0000, Luya Tshimbalanga wrote:
I confirm my system, an AMD/AMD hybrid graphics powered laptop, is affected by executing 'systemctl restart systemd-udev- trigger.service' as root on desktop. The desktop session crashed and the login screen comes afterwards. The same command on a dedicated NVIDIA graphics card is fairly harmless.
It seems the issue is affecting all hardware with hybrid graphics.
That's great data, thanks - I'm gonna find a few other people to test, but it definitely sounds like 'hybrid graphics' is the trigger here.
I have AMD/Intel hybrid graphics and the problem doesn't happen. This is F24, GNOME on Wayland.
On Tue, 2016-10-04 at 15:29 -0600, Chris Murphy wrote:
On Tue, Oct 4, 2016 at 2:39 PM, Adam Williamson adamwill@fedoraproject.org wrote:
On Tue, 2016-10-04 at 20:34 +0000, Luya Tshimbalanga wrote:
I confirm my system, an AMD/AMD hybrid graphics powered laptop, is affected by executing 'systemctl restart systemd-udev- trigger.service' as root on desktop. The desktop session crashed and the login screen comes afterwards. The same command on a dedicated NVIDIA graphics card is fairly harmless.
It seems the issue is affecting all hardware with hybrid graphics.
That's great data, thanks - I'm gonna find a few other people to test, but it definitely sounds like 'hybrid graphics' is the trigger here.
I have AMD/Intel hybrid graphics and the problem doesn't happen. This is F24, GNOME on Wayland.
Well, the bug is an X crash, so it's not surprising it doesn't exist in Wayland. ;) Good job Wayland for not crashing, I guess...
On 04/10/16 16:51, Adam Williamson wrote:
Recently several reports of people getting 'duplicated packages' and 'kernel updates not working' have come through to us in QA from Fedora 24 users. I managed to get one reporter to explain more specifically what happened, and it sounds a lot like what's happening is that something in the 'dnf update' process can cause a GNOME or X crash, possibly depending on hardware or package set installed. When that happens, the update process is killed and does not complete cleanly, which is why you get 'duplicated packages' and other odd results.
I had it happen this morning in fact.
When it happens X crashes and you're left with duplicate package entries that a distro-sync will normally fix though this morning was a bit more fun because systemd was included and the distro-sync didn't want to remove the duplicate.
It only ever happens on one of my machines, so it's obviously something about the environment, possibly the graphics card as that's the only machine I do this on that has nvidia graphics?
The weird thing is that there was nothing particularly X related in the update that caused it this morning. In fact systemd was probably about the most low level thing that was in the update.
Tom
On Tue, 2016-10-04 at 21:54 +0100, Tom Hughes wrote:
It only ever happens on one of my machines, so it's obviously something about the environment, possibly the graphics card as that's the only machine I do this on that has nvidia graphics?
The weird thing is that there was nothing particularly X related in the update that caused it this morning. In fact systemd was probably about the most low level thing that was in the update.
We've pretty much pinned it down, now. The recipe is: hybrid graphics + systemd-udev update == X crash. That is, if there's a systemd-udev update in the dnf transaction, and the system has hybrid graphics, and X is running while the update runs, X will crash. If you're running the update in a VT, X will crash but the update will complete OK. If you run the update inside X, the update process will die when X crashes and your system will be left messed up.
I'm gonna write up a blog post and spread the word about this a bit, I'll send the link shortly; if people could spread it around that'd be great.
The bug reports are https://bugzilla.redhat.com/show_bug.cgi?id=1341327%C2%A0(for the X side) and https://bugzilla.redhat.com/show_bug.cgi?id=1378974%C2%A0(for the systemd side). We are going to try and ensure the bug doesn't affect F25 Beta installs. An update is pending for F24, but ironically (since the service restart is in %postun not %post), the transaction to apply the update will *suffer from* the problem; the update will only ensure that *subsequent* updates no longer trigger the problem.
Thanks for the data everyone!
On 04/10/16 22:04, Adam Williamson wrote:
We've pretty much pinned it down, now. The recipe is: hybrid graphics + systemd-udev update == X crash. That is, if there's a systemd-udev update in the dnf transaction, and the system has hybrid graphics, and X is running while the update runs, X will crash. If you're running the update in a VT, X will crash but the update will complete OK. If you run the update inside X, the update process will die when X crashes and your system will be left messed up.
My machine at work where this happens isn't hybrid. It has three monitors spread over two Nvidia cards though, driven by nouveau rather than the proprietary drivers.
Tom
On Tue, 2016-10-04 at 22:15 +0100, Tom Hughes wrote:
On 04/10/16 22:04, Adam Williamson wrote:
We've pretty much pinned it down, now. The recipe is: hybrid graphics + systemd-udev update == X crash. That is, if there's a systemd-udev update in the dnf transaction, and the system has hybrid graphics, and X is running while the update runs, X will crash. If you're running the update in a VT, X will crash but the update will complete OK. If you run the update inside X, the update process will die when X crashes and your system will be left messed up.
My machine at work where this happens isn't hybrid. It has three monitors spread over two Nvidia cards though, driven by nouveau rather than the proprietary drivers.
OK, so probably strictly speaking this is triggered by having multiple adapters. 'Hybrid graphics' is by far the most common case of that these days, though. Thanks for the note, I'll revise my blog post (that I'm writing ATM).
Am Tue, 04 Oct 2016 14:22:34 -0700 schrieb Adam Williamson adamwill@fedoraproject.org:
On Tue, 2016-10-04 at 22:15 +0100, Tom Hughes wrote:
On 04/10/16 22:04, Adam Williamson wrote:
We've pretty much pinned it down, now. The recipe is: hybrid graphics + systemd-udev update == X crash. That is, if there's a systemd-udev update in the dnf transaction, and the system has hybrid graphics, and X is running while the update runs, X will crash. If you're running the update in a VT, X will crash but the update will complete OK. If you run the update inside X, the update process will die when X crashes and your system will be left messed up.
My machine at work where this happens isn't hybrid. It has three monitors spread over two Nvidia cards though, driven by nouveau rather than the proprietary drivers.
OK, so probably strictly speaking this is triggered by having multiple adapters. 'Hybrid graphics' is by far the most common case of that these days, though. Thanks for the note, I'll revise my blog post (that I'm writing ATM).
I did not see such crashes on my hybrid laptop with F24, X and gnome-shell. I mostly use the intel gpu, but some games run on the nvidia-gpu with nouveau-driver. I tried the "systemctl restart systemd-udev-trigger.service"-approach from a "normal" terminal and from a terminal under nouveau and there was not even a flashing of the monitor visible. I do not have a secondary (or third ...) monitor connected, just the default laptop screen.
Jens
On Tue, 2016-10-04 at 23:54 +0200, Jens Lody wrote:
Am Tue, 04 Oct 2016 14:22:34 -0700 schrieb Adam Williamson adamwill@fedoraproject.org:
On Tue, 2016-10-04 at 22:15 +0100, Tom Hughes wrote:
On 04/10/16 22:04, Adam Williamson wrote:
We've pretty much pinned it down, now. The recipe is: hybrid graphics + systemd-udev update == X crash. That is, if there's a systemd-udev update in the dnf transaction, and the system has hybrid graphics, and X is running while the update runs, X will crash. If you're running the update in a VT, X will crash but the update will complete OK. If you run the update inside X, the update process will die when X crashes and your system will be left messed up.
My machine at work where this happens isn't hybrid. It has three monitors spread over two Nvidia cards though, driven by nouveau rather than the proprietary drivers.
OK, so probably strictly speaking this is triggered by having multiple adapters. 'Hybrid graphics' is by far the most common case of that these days, though. Thanks for the note, I'll revise my blog post (that I'm writing ATM).
I did not see such crashes on my hybrid laptop with F24, X and gnome-shell. I mostly use the intel gpu, but some games run on the nvidia-gpu with nouveau-driver. I tried the "systemctl restart systemd-udev-trigger.service"-approach from a "normal" terminal and from a terminal under nouveau and there was not even a flashing of the monitor visible. I do not have a secondary (or third ...) monitor connected, just the default laptop screen.
OK, that's the second report of an unaffected hybrid system (cmurf said the same thing).
So basically we can say that at least several systems with multiple adapters are affected, and no-one has yet reported a system with a single adapter being affected so far as I can tell. But not *all* multiple adapter systems are affected. We don't know what the difference is yet. Still, safest thing is just to assume that if you have a multiple adapter system you may be affected.
On Tue, 2016-10-04 at 14:04 -0700, Adam Williamson wrote:
I'm gonna write up a blog post and spread the word about this a bit, I'll send the link shortly; if people could spread it around that'd be great.
Blog post:
https://www.happyassassin.net/2016/10/04/x-crash-during-fedora-update-when-s...
If folks can spread that around, it'd be great. Please let me know of any errors or inaccuracies you spot.
Adam Williamson wrote:
If you're using Workstation, the offline update system is expressly designed to minimize the likelihood of this kind of problem, so please do consider using it.
In CentOS or Debian I can afford to reboot for every update. With Fedora's rapid stream of updates that's simply not workable.
Otherwise, at least run 'dnf update' in a VT - hit ctrl-alt-f3 to get a VT console login prompt, log in, and do it there.
In a VT I'll often be unable to review the list of updates before hitting Y, as I'll only see the end of the list.
So if this bug is specific to F24 and multiple GPUs, then I guess I'm lucky to be still using F23 and having only one GPU.
Björn Persson
On Wed, 2016-10-05 at 00:20 +0200, Björn Persson wrote:
Adam Williamson wrote:
If you're using Workstation, the offline update system is expressly designed to minimize the likelihood of this kind of problem, so please do consider using it.
In CentOS or Debian I can afford to reboot for every update. With Fedora's rapid stream of updates that's simply not workable.
Then just don't apply every update. There's no law that says you have to...if there's a pending security update you get a different and more urgent notification, btw.
Otherwise, at least run 'dnf update' in a VT - hit ctrl-alt-f3 to get a VT console login prompt, log in, and do it there.
In a VT I'll often be unable to review the list of updates before hitting Y, as I'll only see the end of the list.
You can look at the list from a desktop terminal but actually run the transaction from the VT, or you can run inside screen or tmux and use their scrollback features...
Adam Williamson wrote:
On Wed, 2016-10-05 at 00:20 +0200, Björn Persson wrote:
Adam Williamson wrote:
If you're using Workstation, the offline update system is expressly designed to minimize the likelihood of this kind of problem, so please do consider using it.
In CentOS or Debian I can afford to reboot for every update. With Fedora's rapid stream of updates that's simply not workable.
Then just don't apply every update. There's no law that says you have to...if there's a pending security update you get a different and more urgent notification, btw.
I don't get any notifications at all. I just try to remember to run Yum periodically, and there I'm not told which updates are security updates.
I used to subscribe to the package-announce list, but I had to drop that when some broken program started sending invalid mails to the list, and the list server kicked me out for rejecting invalid mail. It was never a very good notification mechanism anyway, as I always had to wait a day or two for the packages to appear on the mirrors before I could update.
Many years ago there was sometimes a tray applet that could notify me about pending updates, but it disappeared. If something similar exists now, then I don't know its name so I don't know what to install. Perhaps there is some notification mechanism in Gnome 3; I wouldn't know. I'm not aunt Tillie so I don't use Gnome 3.
Björn Persson
On Wed, Oct 05, 2016 at 01:13:35AM +0200, Björn Persson wrote:
I don't get any notifications at all. I just try to remember to run Yum periodically, and there I'm not told which updates are security updates.
If you install fedora-motd, it'll update /etc/motd with pending update info when you log in over ssh. And, while it's not really documented as such, just running `motdgen` will give you output like:
09:22:04 up 11 days, 22:13, 1 user, load average: 0.03, 0.12, 0.22 Updates Information Summary: available 2 Security notice(s) 1 Bugfix notice(s) 4 Enhancement notice(s)
This is a fairly new thing and could use input and further development.
On Wed, 5 Oct 2016 00:20:25 +0200 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
In a VT I'll often be unable to review the list of updates before hitting Y, as I'll only see the end of the list.
An alternative to Adam's suggestions.
It takes a couple of logins as root, but running dnf update > /tmp/dnf_out 2> /tmp/dnf_ror in one terminal and then doing a less /tmp/dnf_* in another allows a view of the updates before approving or denying them. When you see the (y or N) in the less output, go back to the console running the program and make your selection. A little clumsy, but works fine.
stan wrote:
On Wed, 5 Oct 2016 00:20:25 +0200 Björn Persson Bjorn@xn--rombobjrn-67a.se wrote:
In a VT I'll often be unable to review the list of updates before hitting Y, as I'll only see the end of the list.
An alternative to Adam's suggestions.
It takes a couple of logins as root, but running dnf update > /tmp/dnf_out 2> /tmp/dnf_ror in one terminal and then doing a less /tmp/dnf_* in another allows a view of the updates before approving or denying them. When you see the (y or N) in the less output, go back to the console running the program and make your selection. A little clumsy, but works fine.
It's pretty obvious that improvement is needed when people start to suggest *that* kind of hack-arounds.
Björn Persson
On Tue, Oct 4, 2016 at 4:20 PM, Björn Persson <Bjorn@rombobjörn.se> wrote:
Adam Williamson wrote:
If you're using Workstation, the offline update system is expressly designed to minimize the likelihood of this kind of problem, so please do consider using it.
In CentOS or Debian I can afford to reboot for every update. With Fedora's rapid stream of updates that's simply not workable.
I don't know what the trigger is, but GNOME Software doesn't ask you to do updates more than once a week. If you do 'dnf update' daily, yeah it'll find something to update most of the time.
Once a week is still pretty frequent, but Software doesn't insist that you install the updates once they're available and downloaded. It does it on the next reboot which you can do whenever you'd reboot anyway.
Otherwise, at least run 'dnf update' in a VT - hit ctrl-alt-f3 to get a VT console login prompt, log in, and do it there.
In a VT I'll often be unable to review the list of updates before hitting Y, as I'll only see the end of the list.
It can be piped to more.
This is the first I've heard of any recommendation like this. If running `dnf upgrade` from a graphical console is such a big and well-known risk, then why isn't it mentioned in the dnf documentation? I've posted about this on the dnf Bugzilla.
https://bugzilla.redhat.com/show_bug.cgi?id=1381785
I'm having a hard time finding anything about this in the Fedora Wiki either. If you could recommend any particular reading that could explain this some more, I'd appreciate it.
I tried to read every message in this email thread, but I'm still not clear: It seems the bug that inspired the original post is based on certain graphics hardware, but you still say it's best not to run system updates from a graphical session at all anyway. Is most of the risk related specifically to X and the large software stack that runs on it, or is it simply a problem of numbers, where more running processes means more things could crash while dnf installs updates? Fedora Workstation users are apparently recommended to use GNOME Software's reboot/update feature; what's the recommended way to update all packages on instances of Fedora Server or Fedora Cloud?
OK this is from just one thread, pretty much exactly two years ago. It's a long thread so these are just extractions I think are useful in getting a few different data points about the rationalization of offline updates, and context for the use case where they're most well suited (or not).
And really the bottom line is, dnf update is fine the vast majority of the time, except when it isn't. Failures are somewhere in between a bug and not at all surprising. And people have been working hard on solving this for years.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
There are a bunch of other threads scattered about on multiple lists about and why OSTree. About and why LVM thinp snapshots, Btrfs, and Snapper. Why Chrome OS, Android, and also even CoreOS went with the A/B partitioning layout they did.
On Tue, Oct 4, 2016 at 9:31 PM, Chris Murphy lists@colorremedies.com wrote:
OK this is from just one thread, pretty much exactly two years ago. It's a long thread so these are just extractions I think are useful in getting a few different data points about the rationalization of offline updates, and context for the use case where they're most well suited (or not).
And really the bottom line is, dnf update is fine the vast majority of the time, except when it isn't. Failures are somewhere in between a bug and not at all surprising. And people have been working hard on solving this for years.
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
One more for that thread is good as well...
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On Tue, Oct 4, 2016 at 8:31 PM, Chris Murphy lists@colorremedies.com wrote:
And really the bottom line is, dnf update is fine the vast majority of the time, except when it isn't. Failures are somewhere in between a bug and not at all surprising. And people have been working hard on solving this for years.
Chris, first of all thanks much for posting the links. They basically reinforced my opinion. I have no issue whatsoever with "offline" updates. It is of course a valid approach to a problem. My concern was that the blanket implication about the safety of using DNF within a DE. Even your comment that it is "fine... until it isn't" (which can be said about anything) proves the point.
Packagekit... is "safe until it isn't" - refer to: https://bugzilla.redhat.com/show_bug.cgi?id=1259865 which by the way caused me a bunch of grief because on a lark one day I decided to try it out... sucks to be me I guess.
If offline updates have a place (and yes I believe they do) then why isn't that functionality built into DNF now? I would assume (and yes, I know what happens when people ASS-U-ME - ;-) ) that it is because the DNF team doesn't believe the risk/benefit ratio is high enough to put it in yet and they believe other features/functionality are more beneficial.
That said, they basically already do it with the dnf-system-upgrade plugin; so why not just expand that a bit. Also, while i completely understand that it is much easier to just use a sledgehammer and say "offline upgrades for everything" - we both know that isn't required. Again, there is a place for "offline" updates - and I would like to see that option in DNF - but everything has it's place.
On Wed, Oct 5, 2016 at 6:51 AM, Gerald B. Cox gbcox@bzb.us wrote:
That said, they basically already do it with the dnf-system-upgrade plugin; so why not just expand that a bit. Also, while i completely understand that it is much easier to just use a sledgehammer and say "offline upgrades for everything" - we both know that isn't required. Again, there is a place for "offline" updates - and I would like to see that option in DNF - but everything has it's place.
I created a RFE bug requesting that the upgrade function in DNF be changed to incorporate "offline" upgrades as an option. If it is really an issue, DNF should handle it. https://bugzilla.redhat.com/show_bug.cgi?id=1382063
On Wed, Oct 5, 2016 at 11:59 AM, Gerald B. Cox gbcox@bzb.us wrote:
I created a RFE bug requesting that the upgrade function in DNF be changed to incorporate "offline" upgrades as an option. If it is really an issue, DNF should handle it. https://bugzilla.redhat.com/show_bug.cgi?id=1382063
This would be really nice to have. I'd like to add that at least the MATE and Cinnamon spins, possibly others, do not include PackageKit and instead expect users to update using yumex-dnf or dnf itself. So ideally an offline update mechanism can be added to dnf, and then exposed in yumex-dnf. But we may have to consider installing PackageKit in those spins. My concern with this is that PackageKit and dnf do not share history and many users are used to using dnf.
5.10.2016 22:21 Dan Book grinnz@gmail.com wrote:
[...] I'd like to add that at least the MATE and Cinnamon spins, possibly others, do not include PackageKit and instead expect users to update using yumex-dnf or dnf itself. So ideally an offline update mechanism can be added to dnf, and then exposed in yumex-dnf. But we may have to consider installing PackageKit in those spins. My concern with this is that PackageKit and dnf do not share history and many users are used to using dnf.
What about implementing yumex-pk, I mean another tool forked from yumex but backed by PackageKit, and eventually dropping dnf and leaving PK as the only tool, both commandline and GUI?
I know we have GNOME Software which already does it but in more high level and less power user oriented manner.
Rafal
On Wed, 2016-10-05 at 02:59 +0000, Andrew Toskin wrote:
This is the first I've heard of any recommendation like this. If running `dnf upgrade` from a graphical console is such a big and well-known risk, then why isn't it mentioned in the dnf documentation? I've posted about this on the dnf Bugzilla.
https://bugzilla.redhat.com/show_bug.cgi?id=1381785
I'm having a hard time finding anything about this in the Fedora Wiki either. If you could recommend any particular reading that could explain this some more, I'd appreciate it.
Hmm, well, there isn't necessarily an awful lot of explicit documentation of this, I guess. I dunno what's in the user guides these days. When I said we don't recommend updating from a desktop console, I guess I was thinking mostly of mailing list discussion, IRC, forums etc. rather than formal documentation. I don't think there really *is* any How To Update Your System guide, like we have an Upgrading guide.
I tried to read every message in this email thread, but I'm still not clear: It seems the bug that inspired the original post is based on certain graphics hardware, but you still say it's best not to run system updates from a graphical session at all anyway. Is most of the risk related specifically to X and the large software stack that runs on it, or is it simply a problem of numbers, where more running processes means more things could crash while dnf installs updates?
I'd say broadly speaking both, but the most disruptive and potentially catastrophic effect is when the update process itself crashes or is killed. Because of how RPM transactions work, this generally leaves you with RPM convinced you have two copies of a bunch of packages installed, and cleaning that up is kind of tedious. The more processes are running underneath the dnf process, the more likely the dnf process is to get knocked out by something else. (I don't know if dnf could sensibly be changed to mitigate this issue; it's really not my focus. I just want to try and help real users deal with the software as it currently exists.)
The effect where the update causes running processes to misbehave is usually less of a big deal and more just a case of 'eh, restart it or reboot, no big deal'.
Note I didn't write about using tmux or screen for two reasons:
1) I'm not sure exactly how safe that is from the systemd KillUserProcesses change 2) I didn't want to talk about somewhat-advanced topics, I wanted to keep it simple
but if you do actually know what you're doing with tmux/screen, and someone can clear up the KillUserProcesses thing (or you just make sure the server process is launched outside of the user session, I guess), then I think running an update from a tmux/screen session running on a terminal in a desktop should be nearly as safe as doing it in a VT.
Fedora Workstation users are apparently recommended to use GNOME Software's reboot/update feature; what's the recommended way to update all packages on instances of Fedora Server or Fedora Cloud?
I'm not sure we actually *have* an official recommendation. My take would be that it depends on your risk tolerance. Obviously on a typical Server / Cloud install you're not going to be running the update in a graphical desktop session, so that avoids a lot of the risk right there. It's still technically a bit safer to use the pkcon-driven offline update process than just to update in a VT or with dnf- automatic or a cron job or something, but it's a much smaller difference, I'd say.
The one thing I absolutely would advise against: don't do an update over ssh! Unless you use screen or tmux, of course. But just sshing in and running an update is a great way to potentially hit trouble.
On Tue, Oct 04, 2016 at 20:42:11 -0700, Adam Williamson adamwill@fedoraproject.org wrote:
I'd say broadly speaking both, but the most disruptive and potentially catastrophic effect is when the update process itself crashes or is killed. Because of how RPM transactions work, this generally leaves you with RPM convinced you have two copies of a bunch of packages installed, and cleaning that up is kind of tedious. The more processes are running underneath the dnf process, the more likely the dnf process is to get knocked out by something else. (I don't know if dnf could sensibly be changed to mitigate this issue; it's really not my focus. I just want to try and help real users deal with the software as it currently exists.)
package-cleanup --cleandupes still works on f25, though I am not sure what the proper dnf version of this is. (I needed this about a week ago when an issue triggered by a kernel problem I don't fully understand caused an update to be terminated.) You can usually get the updates to finish and then clean up the duplicates. Sometimes it gets trickier.
The one thing I absolutely would advise against: don't do an update over ssh! Unless you use screen or tmux, of course. But just sshing in and running an update is a great way to potentially hit trouble.
This isn't as bad as you might think. While I mean to use screen, I often forget and very rarely have problems as restarting sshd doesn't shut down existing ssh sessions.
On Wed, 5 Oct 2016 10:56:35 -0500 Bruno Wolff III bruno@wolff.to wrote:
On Tue, Oct 04, 2016 at 20:42:11 -0700, Adam Williamson adamwill@fedoraproject.org wrote:
I'd say broadly speaking both, but the most disruptive and potentially catastrophic effect is when the update process itself crashes or is killed. Because of how RPM transactions work, this generally leaves you with RPM convinced you have two copies of a bunch of packages installed, and cleaning that up is kind of tedious. The more processes are running underneath the dnf process, the more likely the dnf process is to get knocked out by something else. (I don't know if dnf could sensibly be changed to mitigate this issue; it's really not my focus. I just want to try and help real users deal with the software as it currently exists.)
package-cleanup --cleandupes still works on f25, though I am not sure what the proper dnf version of this is. (I needed this about a week ago when an issue triggered by a kernel problem I don't fully understand caused an update to be terminated.) You can usually get the updates to finish and then clean up the duplicates. Sometimes it gets trickier.
From 'man yum2dnf':
Detailed table for package-cleanup replacement:
┌─────────────────────────────┬─────────────────────────────┐ │package-cleanup --dupes │ dnf repoquery --duplicated │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --leaves │ dnf repoquery --unneeded │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --orphans │ dnf repoquery --extras │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --oldkernels │ dnf repoquery --installonly │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --problems │ dnf repoquery --unsatisfied │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --cleandupes │ dnf remove --duplicated │ ├─────────────────────────────┼─────────────────────────────┤ │package-cleanup --oldkernels │ dnf remove --oldinstallonly │ └─────────────────────────────┴─────────────────────────────┘
The one thing I absolutely would advise against: don't do an update over ssh! Unless you use screen or tmux, of course. But just sshing in and running an update is a great way to potentially hit trouble.
This isn't as bad as you might think. While I mean to use screen, I often forget and very rarely have problems as restarting sshd doesn't shut down existing ssh sessions.
Right, thats the thing... 99.9% of the time you are just fine. :)
kevin
On 10/5/2016 9:37 AM, Kevin Fenzi wrote:
On Wed, 5 Oct 2016 10:56:35 -0500 Bruno Wolff III bruno@wolff.to wrote:
On Tue, Oct 04, 2016 at 20:42:11 -0700, Adam Williamson adamwill@fedoraproject.org wrote:
The one thing I absolutely would advise against: don't do an update over ssh! Unless you use screen or tmux, of course. But just sshing in and running an update is a great way to potentially hit trouble.
This isn't as bad as you might think. While I mean to use screen, I often forget and very rarely have problems as restarting sshd doesn't shut down existing ssh sessions.
Right, thats the thing... 99.9% of the time you are just fine. :)
kevin
After a check-update preview, I can count the number of times "yum -y update &" has ever failed me over the years on one hand.
-jc
On Wed, 2016-10-05 at 10:43 -0700, Japheth Cleaver wrote:
After a check-update preview, I can count the number of times "yum -y update &" has ever failed me over the years on one hand.
That still sounds like somewhere between 1 and 5 times too many. =)
The thing is, the case where dnf crashes/dies in the middle of the update is a *really nasty* failure case. It doesn't have to happen very often to be a problem.
Adam Williamson wrote:
I don't want to get in the KDE folks' bad graces, but this likely could also affect KDE's graphical update system, so I'd advise against using that for the present too.
Offline updates in KDE are just not going to happen any time soon.
I also don't see offline updates as a reasonable solution to this problem:
* For this particular crash, the bug should simply be tracked down and fixed.
* As a general workaround for this type of crashes, we need a complete-transaction command in DNF – please add your voices to: https://bugzilla.redhat.com/show_bug.cgi?id=1091702 – and not the sledgehammer approach of doing all updates offline.
Kevin Kofler
On Thu, 2016-10-06 at 00:05 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
I don't want to get in the KDE folks' bad graces, but this likely could also affect KDE's graphical update system, so I'd advise against using that for the present too.
Offline updates in KDE are just not going to happen any time soon.
I also don't see offline updates as a reasonable solution to this problem:
- For this particular crash, the bug should simply be tracked down and fixed.
We already did that, but two things:
1) that doesn't do a damn thing to help anyone who ran into it before we tracked it down and fixed it. Offline updates sure were a great *mitigation* before we fixed the bug.
2) the short-term fix is to remove the service restart from the package scriptlets, but because the service restart was in the %postun scriptlet, the update can't fix it immediately: indeed, *installing* the update will inevitably trigger the bug. The update can only prevent it happening for *subsequent* updates.
- As a general workaround for this type of crashes, we need a complete-transaction command in DNF – please add your voices to: https://bugzilla.redhat.com/show_bug.cgi?id=1091702 – and not the sledgehammer approach of doing all updates offline.
I'm certainly not opposed to improving the experience of recovering from this kind of crash, but I'm really not a big fan of the message 'if the update process happens to crash X, hope you understood what happened, and here's the magic recovery command you can run'.
I honestly think GNOME has this exactly right for the 'general' user: the safe thing to do is to strongly encourage offline updates, i.e. don't offer any online update mechanism through the desktop. In a completely practical sense, given the current state of the tools and the fact that we know bugs like this crop up - not *often*, but more than *never* - I think it's a more responsible approach than running the update process inside the desktop session.
Could KDE perhaps look into making it so the update process runs outside of the desktop session somehow, if it's not going to go to full-on offline updates 'any time soon'? I know this decision has its own complexities, though.
On Wed, Oct 5, 2016 at 3:53 PM, Adam Williamson adamwill@fedoraproject.org wrote:
I honestly think GNOME has this exactly right for the 'general' user: the safe thing to do is to strongly encourage offline updates, i.e. don't offer any online update mechanism through the desktop. In a completely practical sense, given the current state of the tools and the fact that we know bugs like this crop up - not *often*, but more than *never* - I think it's a more responsible approach than running the update process inside the desktop session.
Could KDE perhaps look into making it so the update process runs outside of the desktop session somehow, if it's not going to go to full-on offline updates 'any time soon'? I know this decision has its own complexities, though.
OK, I just think we need to do a reality check here. I initially was trying to find out exactly what was the risk... all the posts I read was more or less 99% of the time you'll be fine, but "be afraid, be very afraid". I then asked for more clarification.... "oh, it is too big to clarify, but be afraid, be very afraid... this is a serious issue". (Oh but by the way, most of the people who claim it is a serious issue go ahead and do online updates anyway...)
OK, I can see the value in offline updates, so I create a RFE to DNF, which I thought was suppose to be THE STANDARD FEDORA SOLUTION. The response, which I completely understood and agreed with was basically, yeah, this is a good idea... but it is a low priority. The implication is that they believe there are more important items which demand their attention.
First of all, if this was such a serious issue, Fedora completely failed in making it known. From what I gather the target audience for "offline" updates was novice users. What follows are rhetorical questions:
Where was the campaign to communicate this "serious issue" to the Fedora community? Do you think you're going to get the message out by posting to the development mailing lists? Where was this discussion when we went from YUM to DNF if it was such an issue? If this is such a huge issue, why doesn't the DNF team consider it a higher priority? Why are we asking that each DE reinvent the wheel on this when we have DNF? That just seems to be a complete waste of resources.
Every software has risks. I have yet to have an issue using YUM or DNF for online updates. The only time I have experienced an issue (which BTW was a complete PITA) was with PackageKit. So much for that...
If Fedora collectively believes this is a serious enough issue then get the DNF team to change their priorities - otherwise people need to consider the risk in association with the exposure. Everything that I have read indicates that it is minimal. That isn't to say it shouldn't be done... what I am saying is people need to stop being alarmist and be worried about more serious issues.
On Wed, 2016-10-05 at 19:17 -0700, Gerald B. Cox wrote:
On Wed, Oct 5, 2016 at 3:53 PM, Adam Williamson adamwill@fedoraproject.org wrote:
I honestly think GNOME has this exactly right for the 'general' user: the safe thing to do is to strongly encourage offline updates, i.e. don't offer any online update mechanism through the desktop. In a completely practical sense, given the current state of the tools and the fact that we know bugs like this crop up - not *often*, but more than *never* - I think it's a more responsible approach than running the update process inside the desktop session.
Could KDE perhaps look into making it so the update process runs outside of the desktop session somehow, if it's not going to go to full-on offline updates 'any time soon'? I know this decision has its own complexities, though.
<snip pointless fifteenth repetition of the same stuff>
OK, I can see the value in offline updates, so I create a RFE to DNF, which I thought was suppose to be THE STANDARD FEDORA SOLUTION.
Hmm, well, it's not really that simple, is it? I mean, if you just landed from Mars and installed Fedora Workstation, you'd have no idea what dnf was. It's not remotely discoverable. I haven't read the user guide in forever so I dunno what that says, but no-one reads the docs anyway. What you'd actually find by poking around is GNOME Software. Which isn't dnf at all. The 'STANDARD FEDORA SOLUTION' for Workstation is offline updates with GNOME Software. The 'standard Fedora solution' for KDE is...well I don't know, because KDE being KDE it ships three different package management GUIs, but it's probably not really dnf either. dnf is the STANDARD FEDORA SOLUTION for Server and Cloud, I guess, but you're quite unlikely to be running it from inside X on either of those.
Where was the campaign to communicate this "serious issue" to the Fedora community?
There wasn't one. I took it upon myself to send out an email because I'd seen enough reports of people getting broken updates - there had been I think seven or eight reports in #fedora at that point, and we had a live reporter in #fedora-qa who was helping us pin down the bug. Next time maybe I'll just say 'screw this' and go play golf instead, if this is the thanks I get for trying to help people out. I've already been told today that I have 'NO FUCKING CLUE', and you're not helping. At this point I'm not sure exactly why I bothered.
Do you think you're going to get the message out by posting to the development mailing lists?
The original mail was CC'ed to users@. I wrote a detailed blog post on this yesterday:
https://www.happyassassin.net/2016/10/04/x-crash-during-fedora-update-when-s...
I posted that to the thread (also CC'ed to users@), and I also tweeted and G+'ed it. Either it or the original email has been picked up by enough news outlets (usually the original email, because of course news outlets don't bother to research follow-ups with actual details, sigh) that at this point my inbox is full of 'haha systemd sucks' flames. Which of course also makes me feel great about things.
Where was this discussion when we went from YUM to DNF if it was such an issue?
This didn't change at all between yum and dnf. yum wasn't written to try and survive its console dying, and neither is dnf. I don't know if changing that was considered.
I mean, again, same question I asked Sam: what exactly is your *goal* here? What are trying to achieve by hounding this thread? We're not sending the Update Police around to your house to tell you how to update your systems. If you've decided that you're okay with the risks and you want to keep running 'dnf update' from a terminal in X I'm not going to *stop* you. You can go right ahead. All I know is we were getting multiple reports of multiple real users getting their systems into messed up states due to this bug, so it seems like a responsible idea to alert people to the issue and suggest that they take steps to avoid it.
If we'd had all the precise details about the cause and the affected hardware at the time I wrote the original email, I'd have included it. In fact if I'd known we were going to get that detail quite soon after I sent the email, I'd have held off and sent a more detailed mail later. But I didn't know that, and I didn't want to sit on the issue for an undetermined amount of time, so I wrote about it. I'm terribly freaking sorry that it seems to have caused you such strife.
If this is such a huge issue, why doesn't the DNF team consider it a higher priority?
I don't know. I'm not the DNF team. Maybe they don't believe many people run it from inside X. Maybe they believe people run it in tmux or screen. I'm a QA monkey. I just deal with bugs. All goddamn day long, I deal with bugs. When there's a sufficiently serious one, I try to tell people about it.
Why are we asking that each DE reinvent the wheel on this when we have DNF? That just seems to be a complete waste of resources.
We're not. The offline update mechanism is all implemented in PackageKit. The graphical package managers in both our main release blocking desktops use PackageKit. They don't have to reinvent any wheels to use the offline update mechanism, they just have to put the appropriate graphical wrappers around the relevant PackageKit bits.
I personally would be rather a lot happier if dnf and all the graphical update tools we cared about shared a common base so they shared history and caches and configuration and locks and stuff properly. But they don't. I can't fix this. Please stop bugging me about it.
<snip more pointless repetition>
Everything that I have read indicates that it is minimal.
I don't understand how you can write this, when we have *multiple people reporting in this very thread* that their systems were affected by exactly this bug. I mean, it's just...strange. We flat out know that quite a lot of hardware, if you run a 'dnf update' in an X terminal right now, is going to have a big problem (because there's a systemd- udev update from a fresh F24 install, and there's a system-udev update in updates-testing right now).
What exactly would you have had me do? Just sit on this and pretend there was no bug and when we saw more and more people come into #fedora and complain about it, tell them the risk was minimal so they should just suck it up and get on with their lives? I mean, seriously, what do you suggest?
That isn't to say it shouldn't be done... what I am saying is people need to stop being alarmist
OK, from now on when I see bugs causing serious real world consequences for multiple real world reporters I'll just shut up and pretend it's not happening, shall I? Wouldn't want to be all 'alarmist' about it.
Adam Williamson wrote:
What exactly would you have had me do? Just sit on this and pretend there was no bug and when we saw more and more people come into #fedora and complain about it, tell them the risk was minimal so they should just suck it up and get on with their lives? I mean, seriously, what do you suggest?
Tell people the actual workaround?
rpm -Uvh --nodeps --nopostun \ http://$MIRROR/updates/testing/25/x86_64/s/systemd-udev-231-8.fc25.x86_64.rp...
--nopostun will disable the broken scriptlet. --nodeps will avoid the hard EVR requirement on the rest of systemd. Then a normal dnf update should fix your deps.
Kevin Kofler
On Thu, 2016-10-06 at 10:18 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
What exactly would you have had me do? Just sit on this and pretend there was no bug and when we saw more and more people come into #fedora and complain about it, tell them the risk was minimal so they should just suck it up and get on with their lives? I mean, seriously, what do you suggest?
Tell people the actual workaround?
rpm -Uvh --nodeps --nopostun \ http://$MIRROR/updates/testing/25/x86_64/s/systemd-udev-231-8.fc25.x86_64.rp...
--nopostun will disable the broken scriptlet. --nodeps will avoid the hard EVR requirement on the rest of systemd. Then a normal dnf update should fix your deps.
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
Adam Williamson wrote:
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
Seeing how this is not the first time a broken %postun script causes unfixable trouble with updating a package, I think RPM really needs a way for a new version of a package to override the old version's %postun. It would be similar to how triggers work, but it should run INSTEAD OF the original %postun, not in addition to it. (As I understand it, postun triggers unfortunately run in addition to normal %postun.)
Kevin Kofler
On Fri, 2016-10-07 at 01:25 +0200, Kevin Kofler wrote:
Adam Williamson wrote:
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
Seeing how this is not the first time a broken %postun script causes unfixable trouble with updating a package, I think RPM really needs a way for a new version of a package to override the old version's %postun. It would be similar to how triggers work, but it should run INSTEAD OF the original %postun, not in addition to it. (As I understand it, postun triggers unfortunately run in addition to normal %postun.)
Yeah, that would be handy for sure. We're kicking around ways to use a %pre to subvert the effect of the %postun in the bug report right now, but having a dedicated mechanism would be handy.
On Thu, 2016-10-06 at 10:18 +0200, Kevin Kofler wrote: Adam Williamson wrote: What exactly would you have had me do? Just sit on this and pretend there was no bug and when we saw more and more people come into #fedora and complain about it, tell them the risk was minimal so they should just suck it up and get on with their lives? I mean, seriously, what do you suggest?
Tell people the actual workaround?
rpm -Uvh --nodeps --nopostun \ http://$MIRROR/updates/testing/25/x86_64/s/systemd-udev-231-8.fc25.x86_64.rp...
--nopostun will disable the broken scriptlet. --nodeps will avoid the hard EVR requirement on the rest of systemd. Then a normal dnf update should fix your deps.
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
On Thu, 2016-10-06 at 08:20 -0700, Adam Williamson wrote:
On Thu, 2016-10-06 at 10:18 +0200, Kevin Kofler wrote: Adam Williamson wrote: What exactly would you have had me do? Just sit on this and pretend there was no bug and when we saw more and more people come into #fedora and complain about it, tell them the risk was minimal so they should just suck it up and get on with their lives? I mean, seriously, what do you suggest?
Tell people the actual workaround?
rpm -Uvh --nodeps --nopostun \ http://$MIRROR/updates/testing/25/x86_64/s/systemd-udev-231-8.fc25.x86_64.rp...
--nopostun will disable the broken scriptlet. --nodeps will avoid the hard EVR requirement on the rest of systemd. Then a normal dnf update should fix your deps.
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
Sorry for the missed quotes; evolution's composer seems to have found itself *yet another* failure mode, where it shows me nicely quoted text but sends something with no quote characters at all...sometimes I wonder if someone is doing all this because they lost a bet, or something.
On Thu, Oct 06, 2016 at 08:20:34AM -0700, Adam Williamson wrote:
On Thu, 2016-10-06 at 10:18 +0200, Kevin Kofler wrote:
rpm -Uvh --nodeps --nopostun \ http://$MIRROR/updates/testing/25/x86_64/s/systemd-udev-231-8.fc25.x86_64.rp...
Huh - that's handy, and I did not actually know --nopostun (something new every day, etc.). It does involve including instructions on how to find the package, though, which would inevitably go stale as it moves from u-t to stable. Still, thanks.
Use "dnf download" or yumdownloader to fetch the package?
As for the KDE part:
Adam Williamson wrote:
The 'standard Fedora solution' for KDE is...well I don't know,
The standard updater on the KDE Spin is plasma-pk-updates. The other tools can also do updates (or at least they claim to be able to), but the applet in the system tray that automatically notifies users of updates (and can also be manually refreshed) and performs them is plasma-pk-updates. The intention is that people should not be firing up a windowed application to do updates manually at all, they should just OK the automatic notification in the system tray (i.e., click on the "Install Updates" button).
because KDE being KDE it ships three different package management GUIs,
There are 3 different package management GUIs not because "KDE is KDE", but because we had to work around at least 2 issues with Apper: 1. a Fedora-specific issue: the PackageKit-hif backend (still) does not implement the APIs to enumerate comps groups that Apper needs (which is the main reason Discover was added, so that people can browse packages – Discover uses AppStream instead of PackageKit for that), and 2. an upstream issue: Apper was not ported to KF5 (there is now an experimental port), and Plasma 4 (libplasma 1) widgets/plasmoids cannot be used in Plasma 5 (libplasma 2), so the standalone applet plasma-pk-updates was written (which is likely to stay because it is already better than the Apper plasmoid ever was, it shows information Apper displays only if you bring up the full UI).
We are looking into replacing Apper with something working better, but right now, the most likely candidate would be dnfdragora + plasma-pk-updates (dnfdragora itself has no applet, and is unlikely to get one because it is a cross-desktop solution using the toolkit abstraction libyui; Mageia, where dnfdragora comes from, is also going to implement plasma-pk-updates), and I am not sure Discover is going to be kicked out, either (some people really believe in AppStream – if it was up to me, Discover would be shown the door as soon as dnfdragora is implemented). Unfortunately, since, unlike the 3 current tools, dnfdragora does not use PackageKit (but dnfdaemon), it will not really mean fewer ways to do updates. We are not going to patch out updating support from the package management applications.
Kevin Kofler
Hi every one,
Reading all ideas about solving issues with upgrading systems from working systems are more or less ideas of ad choc solving some issues or even more or less reinventing the wheel. IMO all those ideas will not solve anything and will only increase total level of entropy. After this will be necessary sooner or later add even more ad choc workarounds and so on ..
I've mention already that some solutions are close to reinventing the wheel. Why? Because they've been solved long time ago. To be more precise *more than decade ago*.
I'm working with rpm (RPM Package Manager) more than two decades (try to execute "rpm -qa --channgelog | grep kloczek" and you can find one on my earliest activities still present in any RH based distributions. I've been maintaining for 3 years PLD which at peak time was withs rpm based distribution with more than 5k src.rpm packages). Initially rpm was huge step forward because it's been formalizing many install/upgrade, uninstall, verifications, building, testing problems under single hood. Especially many things related to building packages have been solved very well. So well that even today only some small improvements time too time needs to be done.
From the beginning of the rpm (from time when it was 100% implemented in perl) compare do SySV packages (used on Solaris and BSD*s) and deb (kind of only improved new skyfold on top of original SySV packaging tools ideas) up to now problem of consistent upgrade never been solved completely. Why? Because man assumption about doing upgrade on working system image/resources is broken by design idea. As long as during upgrade process will be deleted some files still used by working processes or will be reopened by those processes always possibility that those processes will be not able normally used resources or will be trying to use resources from wrong version is relatively high. Whatever could be done on packagemanager are to avoid those icebergs is not enough and will never solve those two fundamental uncertain scenarios.
So why with existing rpm is not possible to solve upgrade dilemmas is probably more or less obvious now. So seems like now is yet another iteration of clashes with rmp limitations only question is how (and by who?) those problems have been already solved? Answer is very simple: those problems have been solved almost *decade ago on Solaris* with introduction two crucial technologies like ZFS (Zeta File System) and IPS (Image Packaging system). These two bits on maintaining system resources are interacting very closely and they cannot be used separately (yes .. atm only).
So how *ALL* upgrade problems have been solved on ideas layer? Very simple: by assumption that system upgrade will never (ever) will be done on working system resource. Someone may scratch his head asking "how it is possible to do upgrade if system resources are not touched?". Answer is that it is not possible to implement this idea adding some functionalities to package management (PM) software. Such operation like upgrade needs to be supported by OS and to be more precise by *FS layer*.
So how problem of consistent upgrade have been solved on Solaris using ZFS and IPS? ZFS has ability to create snapshot of the vol (RO resource) and create on top of the shapshot clone (RW resource). Whole upgrade process consist from few steps: - find volumes which needs to be snapshoted and cloned - create clones - mount clones as separated tree and perform upgrad This part is crucial. If anything wrong will happen during upgrade still working system is not affected. It is possible to observe state of broken upgrade and produce very precise diagnostic data allowing to fix upgrade process on layer of packages. In other words *impact of *during *upgrade* on top of still working system *is* *NULL/ZERO!!!* - when upgrade process is finished grub boot loaded configuration is updated to add new root point from from which updated system system image needs to be booted.
As I wrote two technologies here (together) are crucial here to solve 100% upgrade issues: ZFS and IPS. 3rd minor part is bootloaded. Originally on Solaris 10 was used grub and grub2 on Solaris 11 only simplified whole workflow.
So what is missing here on Linux to implement those idea? To be hones .. not to much which is good :) Only few small bolts and beans are missing :)
On Linux at the moment is available btrfs which provides possibility of RW snapshots (equivalent of ZFS clones). All what needs to be added to this layer is btrfs volume attribute indicating that volume needs to be cloned during upgrade in case of more complicated scenarios. Why? Because automatic discovery may be not enough in cases like mayr database upgrade when part of the u[grade may be some format change which needs to be applied in format for example database files used by some application. If in boot loaded will be possible to have to boot entries allowing to boot from original state from before upgrade and all what was done after upgrade upgrade if post PM upgrade operations applied on top of upgraded software will be cloned as well in case any troubles on this stage. Whole rollback/downgrade procedure will only consist from reboot and choose another BE (Boot Environment) All BE management on solaris is dome over one command beadm. This command is used on cloning existing OS resources manually as well. BE idea is connected to to other small bits like running BE and active BE. Running BE it is BE which is used now and active BE it is BE which will be used automatically if it will be used reboot command without specify BE from which system needs to boot after shutdown.
Another small bit which needs to be sorted is related to install procedures implemented in anaconda and post installation procedures in kernel package. What is missing here? anaconda does not allow now to use /boot on btrfs. It forces use ext3/4. Few weeks ago dysk in my laptop started failing so I've attached new disk replacing CD. Initially I've started replicating whole partitions layout as it was applied by anaconda installer with one partition for swap, second one for ext3/4 /boot and / on btrfs. When I've done and after start "btrfs send | btr receive" commands I've found out that in kernel space are loaded ext modules ad they are used only by /boot. So I've stopped everything to change to have only swap partition (without LVM) and btrfs root pool. After copy all resources and generate proper boot loaded on new disk everything still is working so there is no any technical reasons now to have /boot separated!!! Only obstacle is that implemented in kernel package post installation procedure does not like btrfs on /noot and does not update grub boot entries so after few one or two kernel upgrades from rawhide I found that my grub menu is shorter and rhorer :) All what needs to be done to fix this issue is execute "grub2-mkconfig -o /boot/grub2/grub.cfg" I'm pretty sure that above will not break booting from other FSeses :)
Going to the end of his long email ..
All that needs to have done solving all upgrade issues on top of the Fedora in some minimalistic scenario is: - add to dnf BEs management - switch btrfs as default FS - adapt kernel post installation procedure On top of above can be added few other small bits making whole BE management consistent from point of view BEs management.
Anyone who will choose other than btrfs FS will need to accept that it will be more or less dealing with limitations of the calassic more than 30 years old ideas of using non-shapshotable/cloneable volumes and limitations of old SySV packages ideas.
rpm need to die sooner or later as well and probably best would be adapt IPS :) IPS is fully OSS (https://java.net/projects/ips/sources/pkg-gate/show/) and many people see far been thinking about porting it of Linux. However lack of enough stable btrfs was main obstacle. As now btrfs is quite stable IMO it is time to start thinking about move away from rpm as well. However no rush :) As I said and I think that I've prove above that now IPS is not essential maybe another time I'll try to write longer comment why rpm is already dead :)
kloczek
On Thu, Oct 6, 2016 at 7:43 AM, Tomasz Kłoczko kloczko.tomasz@gmail.com wrote:
So how problem of consistent upgrade have been solved on Solaris using ZFS and IPS? ZFS has ability to create snapshot of the vol (RO resource) and create on top of the shapshot clone (RW resource). Whole upgrade process consist from few steps:
- find volumes which needs to be snapshoted and cloned
- create clones
- mount clones as separated tree and perform upgrad
This is what OSTree is effectively doing now.
/usr is read only. And updates aren't ever applied to the current file system tree. A new tree is created and updated, the bootloader configuration is changed, and the system is rebooted (at the user's leisure) to get the updated system. i.e. it makes it clear your system is not running updates until it's rebooted, which it maybe probably who knows.
rpm-ostree integrates RPM support into OSTree.
This part is crucial. If anything wrong will happen during upgrade still working system is not affected. It is possible to observe state of broken upgrade and produce very precise diagnostic data allowing to fix upgrade process on layer of packages. In other words impact of during upgrade on top of still working system is NULL/ZERO!!!
Yes, it's the same for rpm-ostree, i.e. the basis for the various Atomic Host products Fedora creates. And there's going to be a Workstation version of it for Fedora 25 (for very early adopters, it won't be at getfedora.org).
On Linux at the moment is available btrfs which provides possibility of RW snapshots (equivalent of ZFS clones). All what needs to be added to this layer is btrfs volume attribute indicating that volume needs to be cloned during upgrade in case of more complicated scenarios.
It's actually kinda complicated to handle the changes that occur from the time the current tree is snapshot, the update and reboot happens. The current root tree can have changes to /var and /etc happening during the update and before reboot, so those need to get merged such that you can reboot either the old or new tree and have log and journal continuity, and so on. rpm-ostree does this.
Another small bit which needs to be sorted is related to install procedures implemented in anaconda and post installation procedures in kernel package. What is missing here? anaconda does not allow now to use /boot on btrfs. It forces use ext3/4.
It's a grubby bug. It doesn't understand Btrfs subvolumes. Any brave person who wants to fix grubby? Someone else already tried, so there's code available to look at to help grok the logic of what needs to be done; but needs to be implemented in a different way to get approved by upstream. https://bugzilla.redhat.com/show_bug.cgi?id=864198
Can anyone answer this relatively simple question: "Why grubby?" I've seen a number of discussions on various topics surrounding the boot loader that all seem to devolve into "We would love to support that, but grubby doesn't, so we can't."
At what point does the maintenance burden of using grubby outweigh its own benefits?
I don't ask this rhetorically, or because I particularly want to see grubby gone. I just don't see the benefit that we get from having grubby when other distros seem to get by just fine without it, or if they do use it, it doesn't seem to be getting in their way.
Cheers! Eric
On Oct 6, 2016, at 12:12, Chris Murphy lists@colorremedies.com wrote:
On Thu, Oct 6, 2016 at 7:43 AM, Tomasz Kłoczko kloczko.tomasz@gmail.com wrote:
So how problem of consistent upgrade have been solved on Solaris using ZFS and IPS? ZFS has ability to create snapshot of the vol (RO resource) and create on top of the shapshot clone (RW resource). Whole upgrade process consist from few steps:
- find volumes which needs to be snapshoted and cloned
- create clones
- mount clones as separated tree and perform upgrad
This is what OSTree is effectively doing now.
/usr is read only. And updates aren't ever applied to the current file system tree. A new tree is created and updated, the bootloader configuration is changed, and the system is rebooted (at the user's leisure) to get the updated system. i.e. it makes it clear your system is not running updates until it's rebooted, which it maybe probably who knows.
rpm-ostree integrates RPM support into OSTree.
This part is crucial. If anything wrong will happen during upgrade still working system is not affected. It is possible to observe state of broken upgrade and produce very precise diagnostic data allowing to fix upgrade process on layer of packages. In other words impact of during upgrade on top of still working system is NULL/ZERO!!!
Yes, it's the same for rpm-ostree, i.e. the basis for the various Atomic Host products Fedora creates. And there's going to be a Workstation version of it for Fedora 25 (for very early adopters, it won't be at getfedora.org).
On Linux at the moment is available btrfs which provides possibility of RW snapshots (equivalent of ZFS clones). All what needs to be added to this layer is btrfs volume attribute indicating that volume needs to be cloned during upgrade in case of more complicated scenarios.
It's actually kinda complicated to handle the changes that occur from the time the current tree is snapshot, the update and reboot happens. The current root tree can have changes to /var and /etc happening during the update and before reboot, so those need to get merged such that you can reboot either the old or new tree and have log and journal continuity, and so on. rpm-ostree does this.
Another small bit which needs to be sorted is related to install procedures implemented in anaconda and post installation procedures in kernel package. What is missing here? anaconda does not allow now to use /boot on btrfs. It forces use ext3/4.
It's a grubby bug. It doesn't understand Btrfs subvolumes. Any brave person who wants to fix grubby? Someone else already tried, so there's code available to look at to help grok the logic of what needs to be done; but needs to be implemented in a different way to get approved by upstream. https://bugzilla.redhat.com/show_bug.cgi?id=864198
-- Chris Murphy _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Thu, 2016-10-06 at 17:05 -0400, Eric Griffith wrote:
Can anyone answer this relatively simple question: "Why grubby?" I've seen a number of discussions on various topics surrounding the boot loader that all seem to devolve into "We would love to support that, but grubby doesn't, so we can't."
At what point does the maintenance burden of using grubby outweigh its own benefits?
I don't ask this rhetorically, or because I particularly want to see grubby gone. I just don't see the benefit that we get from having grubby when other distros seem to get by just fine without it, or if they do use it, it doesn't seem to be getting in their way.
The major reason it exists, AIUI, is that it provides a consistent interface across different bootloaders (we use bootloaders other than grub on non-Intel architectures). It's also I think mostly consistent across grub-legacy and grub2. This is a big thing for big RHEL sites that want to have consistent bootloader management across disparate arches and RHEL releases (inc. old ones that still use grub-legacy).
The same situation does exist for Fedora, since we *do* have PPC and s390 and stuff as secondary arches.
On Thu, 6 Oct 2016 17:05:45 -0400 Eric Griffith egriffith92@gmail.com wrote:
Can anyone answer this relatively simple question: "Why grubby?" I've seen a number of discussions on various topics surrounding the boot loader that all seem to devolve into "We would love to support that, but grubby doesn't, so we can't."
At what point does the maintenance burden of using grubby outweigh its own benefits?
I don't ask this rhetorically, or because I particularly want to see grubby gone. I just don't see the benefit that we get from having grubby when other distros seem to get by just fine without it, or if they do use it, it doesn't seem to be getting in their way.
Well, I don't know the full history here, but IMHO, the problem is that the way grub2 does config is not very ideal. Most applications when you want to add some new configuration allow you to just do that and leave everything else you already set the way you wanted it alone. With grub2 it expects to completely regenerate it's config file every time you want to add a new entry.
From a practical standpoint this means if you have several entries that work just fine and add a new one you could end up with none of them working instead of just the most recent one failing and allowing you to go back and use one of the previous (working) ones.
Perhaps other distros have figured out better ways to deal with this, I don't know. If someone wanted to go and survey this and report back that information might be of help.
kevin
On Thu, Oct 6, 2016 at 3:17 PM, Kevin Fenzi kevin@scrye.com wrote:
On Thu, 6 Oct 2016 17:05:45 -0400 Eric Griffith egriffith92@gmail.com wrote:
Can anyone answer this relatively simple question: "Why grubby?" I've seen a number of discussions on various topics surrounding the boot loader that all seem to devolve into "We would love to support that, but grubby doesn't, so we can't."
At what point does the maintenance burden of using grubby outweigh its own benefits?
I don't ask this rhetorically, or because I particularly want to see grubby gone. I just don't see the benefit that we get from having grubby when other distros seem to get by just fine without it, or if they do use it, it doesn't seem to be getting in their way.
Well, I don't know the full history here, but IMHO, the problem is that the way grub2 does config is not very ideal.
It's not. And OS discovery is fraught with problems also.
One alternative is Bootloader Spec drop-in scripts. Fedora's GRUB carries a patch to support them in the blscfg.mod, but it implements it in an early interpretation of the spec. rpm-ostree creates drop in scripts per this spec, but instead of depending on blscfg.mod to use them directly, rpm-ostree calls a helper to use the scripts as source material to generate new grub.cfg or extlinux.conf (both bootloaders are supported). rpm-ostree doesn't use grubby at all.
Perhaps other distros have figured out better ways to deal with this, I don't know. If someone wanted to go and survey this and report back that information might be of help.
I don't have a complete or recent evaluation but as of a couple years ago, before Gene Czarcinski passed away, we found no other distribution using grubby. Distros we ran into use grub-mkconfig as recommended by upstream to obliterate the existing grub.cfg and create an entirely new one.
I'm just thinking out loud here, but, given that rpm-ostree does not use grubby, and we do have the Bootloader Spec, and no other distro uses grubby, would it be prudent to take a really hard look at whether grubby is still a path we want to walk?
If it is, then more work obviously needs to be put into it to get it where we want/need it. Personally, I would love to see btrfs subvolume support, so that we could have snapshotting like on Suse, though it appears rpm-ostree would negate the need for it, correct?
If it's not a path we want to walk anymore, then let's announce an "Intention To Deprecate" to give all interested parties (RHEL) plenty of warning, and start figuring out what all will break by the removal of grubby.
Adam, the only other distro that has serious alternate architecture support, AFAIK, is Debian. How do they handle grub2 + non-x86? Likewise, the alternate architectures that we support, how do we handle their bootloaders? Are they grub-based? Ext/Syslinux based? Grub-legacy?
I agree with Kevin that grub2 is.... nonintuitive. But the only other serious option we have is bootctl, and I am not sure if that's even a possibility for a distro like Fedora. I know Arch has it as an install time option, but I don't know it's limitations.
Cheers! Eric
On Oct 6, 2016, at 21:20, Chris Murphy lists@colorremedies.com wrote:
On Thu, Oct 6, 2016 at 3:17 PM, Kevin Fenzi kevin@scrye.com wrote: On Thu, 6 Oct 2016 17:05:45 -0400 Eric Griffith egriffith92@gmail.com wrote:
Can anyone answer this relatively simple question: "Why grubby?" I've seen a number of discussions on various topics surrounding the boot loader that all seem to devolve into "We would love to support that, but grubby doesn't, so we can't."
At what point does the maintenance burden of using grubby outweigh its own benefits?
I don't ask this rhetorically, or because I particularly want to see grubby gone. I just don't see the benefit that we get from having grubby when other distros seem to get by just fine without it, or if they do use it, it doesn't seem to be getting in their way.
Well, I don't know the full history here, but IMHO, the problem is that the way grub2 does config is not very ideal.
It's not. And OS discovery is fraught with problems also.
One alternative is Bootloader Spec drop-in scripts. Fedora's GRUB carries a patch to support them in the blscfg.mod, but it implements it in an early interpretation of the spec. rpm-ostree creates drop in scripts per this spec, but instead of depending on blscfg.mod to use them directly, rpm-ostree calls a helper to use the scripts as source material to generate new grub.cfg or extlinux.conf (both bootloaders are supported). rpm-ostree doesn't use grubby at all.
Perhaps other distros have figured out better ways to deal with this, I don't know. If someone wanted to go and survey this and report back that information might be of help.
I don't have a complete or recent evaluation but as of a couple years ago, before Gene Czarcinski passed away, we found no other distribution using grubby. Distros we ran into use grub-mkconfig as recommended by upstream to obliterate the existing grub.cfg and create an entirely new one.
-- Chris Murphy _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org
On Fri, 7 Oct 2016 00:18:21 -0400 Eric Griffith egriffith92@gmail.com wrote:
I'm just thinking out loud here, but, given that rpm-ostree does not use grubby, and we do have the Bootloader Spec, and no other distro uses grubby, would it be prudent to take a really hard look at whether grubby is still a path we want to walk?
Well, the bootloader spec has a lot of things missing that we expect/want to support. Like multibooting, BIOS boot chainloading booting, etc.
Matthew Garrett started working on proposed changes to the spec: https://www.freedesktop.org/wiki/MatthewGarrett/BootLoaderSpec/ but the owners of the spec didn't seem to interested in those cases and Matthew went on to other work.
So, I think we would need to get buyin to get our changes into the spec and get everyone to agree on it before we would want to move to it.
kevin
On Wed, Oct 5, 2016 at 8:43 PM, Adam Williamson adamwill@fedoraproject.org wrote:
OK, from now on when I see bugs causing serious real world consequences for multiple real world reporters I'll just shut up and pretend it's not happening, shall I? Wouldn't want to be all 'alarmist' about it.
Adam... those questions in my last message were rhetorical. I wasn't expecting an answer.
You're still missing my point.
I appreciate the fact that you took positive action in regards to the systemd/scriptlet bug. The Issue I had was when you made the blanket statement in regards to running DNF. That was the overreach. It was then compounded IMHO when you started the with the implication that the GNOME model should be expanded into other DEs. As you yourself mentioned, you want to help people with this bug - but you expanded that into a mini-crusade for "offline updates".
As Kevin pointed out, the best solution would be to give people the actual workaround.
I would consider the concept of "offline updates" to be a sea change in the mind of most Linux users. I can imagine the press release:
"Ten Years in the making: Fedora develops innovative methodology for safer system updates: REBOOT"
Now of course that can be spun many different ways, but honestly you can guess how that is going to play out in the twitter-verse.
As I mentioned several times, I do understand the value for "offline" updates. I also mentioned that in the RFE Bug: A nice goal would be for DNF to be able to automatically recognize which updates require a reboot and segregate them off for later processing while continuing with those which are safe to do in an online environment. Many people are going to have an issue with reboot for all updates - and as well they should. It just isn't necessary. Which is why I suggested it be added as an option, at least until the logic to handle it automatically could be developed.
Regarding your comment about sharing history, etc. between DNF and PK, it is apparently being addressed: https://lists.fedoraproject.org/pipermail/devel/2015-October/216136.html
If people feel strongly that not doing offline updates is a huge risk then they should escalate and take it up with FESCo. I suspect the reason that the DNF team isn't giving it a high priority is because they don't view it as an urgent issue/risk.
Adam Williamson wrote:
Next time maybe I'll just say 'screw this' and go play golf instead, if this is the thanks I get for trying to help people out.
Others have thanked you, actually, but parts of this email chain have gotten a little heated, so I guess it's worth repeating: Thanks for starting this discussion. I learned something, and I'm not the only one :)
Kevin Kofler writes:
Adam Williamson wrote:
I don't want to get in the KDE folks' bad graces, but this likely could also affect KDE's graphical update system, so I'd advise against using that for the present too.
Offline updates in KDE are just not going to happen any time soon.
I also don't see offline updates as a reasonable solution to this problem:
- For this particular crash, the bug should simply be tracked down and fixed.
Thank you. I'm glad to see that I have not lost my sanity.
- As a general workaround for this type of crashes, we need a complete-transaction command in DNF – please add your voices to: https://bugzilla.redhat.com/show_bug.cgi?id=1091702 – and not the sledgehammer approach of doing all updates offline.
CLOSED/WONTFIX since 2014. No comment.
On Wed, 2016-10-05 at 23:54 -0400, Sam Varshavchik wrote:
- As a general workaround for this type of crashes, we need a
complete-transaction command in DNF – please add your voices to: https://bugzilla.redhat.com/show_bug.cgi?id=1091702 – and not the sledgehammer approach of doing all updates offline.
CLOSED/WONTFIX since 2014. No comment.
Except there is a comment. This is what Ales wrote when closing the bug:
"Closign this as WONTFIX with a vote, i.e. once enough people are CCed in the bug, we will see about adding this."
IOW, he didn't exactly mean WONTFIX. So, if you want that feature added...CC yourself.
Adam Williamson writes:
On Wed, 2016-10-05 at 23:54 -0400, Sam Varshavchik wrote:
- As a general workaround for this type of crashes, we need a
complete-transaction command in DNF – please add your voices to: https://bugzilla.redhat.com/show_bug.cgi?id=1091702 – and not the sledgehammer approach of doing all updates offline.
CLOSED/WONTFIX since 2014. No comment.
Except there is a comment. This is what Ales wrote when closing the bug:
"Closign this as WONTFIX with a vote, i.e. once enough people are CCed in the bug, we will see about adding this."
IOW, he didn't exactly mean WONTFIX. So, if you want that feature added...CC yourself.
That's fair enough.
But I'm still waiting for a logical explanation why setsid() plus sigaction() for selected signal won't be, at least, an interim fix, insofar as preventing a failed install due to an X crashing for some reason, due to a bad scriptlet, or something.
The only proposed explanation is that you still won't immediately know if the transaction is still running or not; without a clear explanation why ps(1) will be insufficient to make that determination. And, of course, implicit in that argument is that setsid()+sigaction() *will* be sufficient and that's just a different problem to solve. But just because the second problem needs a different solution doesn't mean that the first problem cannot be solved.