Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
To recap from those:
- xfce4-session crash: https://bugzilla.redhat.com/show_bug.cgi?id=891113 (work around clearing session cache, it's not shown up again)
- libselinux blow up. This could have caused folks to not be able to boot until upgraded/downgraded. Broken package only in for one day.
- libcdio abi bump with a few broken packages. Fixed in one day.
- xulrunner/firefox broken dep for a day.
- colord multilib issue. Fixed in one day.
- iwlwifi slow. Not a stopper, just anoying. (bug 863424)
- gtk3 update makes nm-applet unhappy (bug https://bugzilla.gnome.org/show_bug.cgi?id=691911 )
- libnl3 abi break. fixed in 2 days.
- mdadm needs newer dracut. Just needs a rawhide build, should be fixed tomorrow.
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
I know some folks at fudcon were talking about rawhide improvements too, so hopefully we will see those proposed before too long as well.
I've tried to suggest things that won't disrupt maintainers much while making things nicer for consumers. Some of these things I have already discussed with folks involved and we are going to try and do them, others are needing more fleshing out/discussion.
I'm very open to more ideas or thoughts, or comments/people willing to work on items from the above wiki page.
Thoughts?
kevin
On 01/23/2013 11:55 AM, Kevin Fenzi wrote:
Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
To recap from those:
xfce4-session crash: https://bugzilla.redhat.com/show_bug.cgi?id=891113 (work around clearing session cache, it's not shown up again)
libselinux blow up. This could have caused folks to not be able to boot until upgraded/downgraded. Broken package only in for one day.
libcdio abi bump with a few broken packages. Fixed in one day.
xulrunner/firefox broken dep for a day.
colord multilib issue. Fixed in one day.
iwlwifi slow. Not a stopper, just anoying. (bug 863424)
gtk3 update makes nm-applet unhappy (bug https://bugzilla.gnome.org/show_bug.cgi?id=691911 )
libnl3 abi break. fixed in 2 days.
mdadm needs newer dracut. Just needs a rawhide build, should be fixed tomorrow.
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
I know some folks at fudcon were talking about rawhide improvements too, so hopefully we will see those proposed before too long as well.
I've tried to suggest things that won't disrupt maintainers much while making things nicer for consumers. Some of these things I have already discussed with folks involved and we are going to try and do them, others are needing more fleshing out/discussion.
I'm very open to more ideas or thoughts, or comments/people willing to work on items from the above wiki page.
Thoughts?
kevin
Would issues with specific 3.8 kernel drivers affecting rawhide be a candidate?
TIA
On Wed, 23 Jan 2013 12:08:21 -0500 "Clyde E. Kunkel" clydekunkel7734@verizon.net wrote:
Would issues with specific 3.8 kernel drivers affecting rawhide be a candidate?
Candidate for a tracker bug? Perhaps. It would depend on how widespread the issue is. Definitely do file a bug on it, and if you think it's a common hardware, notify the list about it so other people can see if they have the same issue.
kevin
On 01/23/2013 12:25 PM, Kevin Fenzi wrote:
On Wed, 23 Jan 2013 12:08:21 -0500 "Clyde E. Kunkel" clydekunkel7734@verizon.net wrote:
Would issues with specific 3.8 kernel drivers affecting rawhide be a candidate?
Candidate for a tracker bug? Perhaps. It would depend on how widespread the issue is. Definitely do file a bug on it, and if you think it's a common hardware, notify the list about it so other people can see if they have the same issue.
kevin
Bug 901828 - radeon 0000:01:00.0: GPU lockup CP stall for more than 10000msec
This is ATI RV630 [Radeon HD 2600 Series] graphics card. Google search shows some issues with the driver in the 3.8 series of kernels.
All other flavors of fedora (non-3.8 kernels) are fine.
2013/1/23 Kevin Fenzi kevin@scrye.com:
Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
[...]
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
I know some folks at fudcon were talking about rawhide improvements too, so hopefully we will see those proposed before too long as well.
I've tried to suggest things that won't disrupt maintainers much while making things nicer for consumers. Some of these things I have already discussed with folks involved and we are going to try and do them, others are needing more fleshing out/discussion.
I'm very open to more ideas or thoughts, or comments/people willing to work on items from the above wiki page.
Thoughts?
Out of curiosity, are you experiencing this, or am I just unlucky? https://bugzilla.redhat.com/show_bug.cgi?id=894623 (I did not debug it and am using a f18 computer for local runs of fedora-review or mock). Otherwise, I noticed rawhide appears to be better tested now, and I did no longer see consecutive non booting upgrades, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=789285 Other breakages I noticed are usually small or short lived, most drastic one was firefox not working for a small window of time.
kevin
-- devel mailing list devel@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/devel
Paulo
On Wed, 23 Jan 2013 15:11:11 -0200 Paulo César Pereira de Andrade paulo.cesar.pereira.de.andrade@gmail.com wrote:
Out of curiosity, are you experiencing this, or am I just unlucky? https://bugzilla.redhat.com/show_bug.cgi?id=894623 (I did not debug it and am using a f18 computer for local runs of fedora-review or mock).
I have not hit that, but I haven't done any mock builds recently.
To debug these kinds of things, it helps to:
- Look at when it last worked. - See if mock or anything mock uses was upgraded - Download old versions from koji and downgrade until it starts working. Then you can note that on the bug and it becomes much easier to isolate.
Otherwise, I noticed rawhide appears to be better tested now, and I did no longer see consecutive non booting upgrades, e.g. https://bugzilla.redhat.com/show_bug.cgi?id=789285 Other breakages I noticed are usually small or short lived, most drastic one was firefox not working for a small window of time.
Yeah, we are doing somewhat better, but we could still do better yet, IMHO.
kevin
On Wed, Jan 23, 2013 at 09:55:07 -0700, Kevin Fenzi kevin@scrye.com wrote:
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
Would the rawhide tracker bug be for rawhide users? I'm guessing the intent would be to make finding out if someone else already filed a bug for a rawhide issue one is experiencing a bit easier?
Manually signing rawhide builds seems to conflict with your anti-goals of not taking away resources from other things. It's probably better to just wait for auto-signing.
Holding builds with broken deps isn't going to help testing in a lot of cases. Sometimes dependent packages take a long time to get rebuilt. For myself I usually remove the packages blocking updates in order to be able to use the new packages. I think a better solution here is to have more proven packagers jump in and do rebuilds for soname bumps. (Unless a package has changes in master that haven't been used for a build yet, this is pretty safe. Though this might run up against people doing builds in branched and not rawhide.)
The anaconda guys have not been big fans of making rawhide installable since it tends to generate bug reports for things that aren't finished yet. (That they already know about.) It may be that install testing is best done with branched.
I find that yum plugin local tends to suck up a lot of space for little benefit. I tend to want a few specific packages to either downgrade or to get fixes that won't show up until the next day. I grab these from koji and add them to my own local repos.
I'd like to see a stronger push to have packagers start work in master, not branched. Inheritance ends up leaving stuff broken longer in rawhide and doesn't test building packages in rawhide, so that problems go unnoticed.
The one day delay may not work that well. If no one is trying the latest kernel, dracut or systemd, then people are still going to get hit, just one day later. Still it would be nice to be able to fix seriously broken stuff in less than one day in a way that's easy for people to use, rather than everyone doing their own manual fix.
On Wed, 23 Jan 2013 11:43:00 -0600 Bruno Wolff III bruno@wolff.to wrote:
On Wed, Jan 23, 2013 at 09:55:07 -0700, Kevin Fenzi kevin@scrye.com wrote:
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
Would the rawhide tracker bug be for rawhide users? I'm guessing the intent would be to make finding out if someone else already filed a bug for a rawhide issue one is experiencing a bit easier?
Well, it's open to discussion. ;)
I could see it being useful for:
- users to watch and see when new bugs are added to it that affect a significant number of rawhide users. If you see this before you update to the broken package you could work around or exclude it until it's finished. ie, increase awareness.
- It could also reduce duplicates.
- Might help maintainers watch for any kinds of 'fallout' on something they landed that may not at first be filed against their package.
Manually signing rawhide builds seems to conflict with your anti-goals of not taking away resources from other things. It's probably better to just wait for auto-signing.
Well, it's easy to do and it only would affect Dennis and Me, so I think it's worth doing since we can do it almost immediately.
It's pretty unclear when we might have an acceptable auto-signing setup.
Holding builds with broken deps isn't going to help testing in a lot of cases. Sometimes dependent packages take a long time to get rebuilt. For myself I usually remove the packages blocking updates in order to be able to use the new packages. I think a better solution here is to have more proven packagers jump in and do rebuilds for soname bumps. (Unless a package has changes in master that haven't been used for a build yet, this is pretty safe. Though this might run up against people doing builds in branched and not rawhide.)
well, sure, but sometimes more than a rebuild is needed.
The anaconda guys have not been big fans of making rawhide installable since it tends to generate bug reports for things that aren't finished yet. (That they already know about.) It may be that install testing is best done with branched.
Perhaps. I would love to have them chime in and see if this has changed any. I think it would be useful to have such testing before branched.
I find that yum plugin local tends to suck up a lot of space for little benefit. I tend to want a few specific packages to either downgrade or to get fixes that won't show up until the next day. I grab these from koji and add them to my own local repos.
Yeah, I'm afraid I agree.
I'd like to see a stronger push to have packagers start work in master, not branched. Inheritance ends up leaving stuff broken longer in rawhide and doesn't test building packages in rawhide, so that problems go unnoticed.
yep. This is one reason I would prefer to just drop the inheritance.
The one day delay may not work that well. If no one is trying the latest kernel, dracut or systemd, then people are still going to get hit, just one day later. Still it would be nice to be able to fix seriously broken stuff in less than one day in a way that's easy for people to use, rather than everyone doing their own manual fix.
There would need to be a way to choose between 'one day delay' and 'today's now'. So, you would get the early adopters testing and fixing things so the people one day behind would get the fix.
For that matter, we could start doing 2 rawhide composes a day, spreading our the changes and allowing fixes in faster.
kevin
On Wed, Jan 23, 2013 at 11:01:50 -0700, Kevin Fenzi kevin@scrye.com wrote:
yep. This is one reason I would prefer to just drop the inheritance.
How would that work at branch? Would we always do a mass rebuild at branch? Would we use a special tag of packages at branch and inherit them, but not later changes?
Le Mer 23 janvier 2013 19:01, Kevin Fenzi a écrit :
On Wed, 23 Jan 2013 11:43:00 -0600 Bruno Wolff III bruno@wolff.to wrote:
On Wed, Jan 23, 2013 at 09:55:07 -0700, Kevin Fenzi kevin@scrye.com wrote:
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
There is a low hanging fruit to make rawhide more pleasant for users : fix firefox/xulrunner dependency locking so yum --skip-broken does not update xulrunner before the corresponding firefox is available.
The firefox breakage you noticed is unfortunately a regular rawhide occurrence, not a one-off.
On Wed, Jan 23, 2013 at 11:01:50AM -0700, Kevin Fenzi wrote:
On Wed, 23 Jan 2013 11:43:00 -0600 Bruno Wolff III bruno@wolff.to wrote:
I'd like to see a stronger push to have packagers start work in master, not branched. Inheritance ends up leaving stuff broken longer in rawhide and doesn't test building packages in rawhide, so that problems go unnoticed.
yep. This is one reason I would prefer to just drop the inheritance.
Definitely agree with this. Additionally inheritance is very confusing.
Slightly off topic but libguestfs often hits Rawhide problems, since we boot and run a small Rawhide appliance in the middle of our builds.
Rich.
On Wed, Jan 23, 2013 at 09:55:07AM -0700, Kevin Fenzi wrote:
Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
So, I've been collecting ideas on how to improve things in rawhide and have made a wiki page for these ideas.
https://fedoraproject.org/wiki/Rawhide_improvement_project_2013
I know some folks at fudcon were talking about rawhide improvements too, so hopefully we will see those proposed before too long as well.
I've tried to suggest things that won't disrupt maintainers much while making things nicer for consumers. Some of these things I have already discussed with folks involved and we are going to try and do them, others are needing more fleshing out/discussion.
I'm very open to more ideas or thoughts, or comments/people willing to work on items from the above wiki page.
If the deps provided by a new build have changed, then verify that the change deps don't cause breakage. If they do, then do not allow the new build into the rawhide target. Queue the build in some kind of build specific 'pending' target. Let all the broken deps be fixed in that pending target, and then atomically merge that target back into rawhide target. Basically at no time ever, should you allow broken deps to make it into rawhide. As long as our build process allows for broken deps, then rawhide is going to be a trainwreck of some kind or another. Requiring people to pre-announce deps breakage to let people promptly rebuild has unequivocably failed to solve the broken deps problems. We need to have a way to deal with this in our build tool chain permanently without humans as the failure point.
Regards, Daniel
Le mercredi 23 janvier 2013 à 20:02 +0000, Daniel P. Berrange a écrit :
If the deps provided by a new build have changed, then verify that the change deps don't cause breakage. If they do, then do not allow the new build into the rawhide target. Queue the build in some kind of build specific 'pending' target. Let all the broken deps be fixed in that pending target, and then atomically merge that target back into rawhide target. Basically at no time ever, should you allow broken deps to make it into rawhide. As long as our build process allows for broken deps, then rawhide is going to be a trainwreck of some kind or another. Requiring people to pre-announce deps breakage to let people promptly rebuild has unequivocably failed to solve the broken deps problems. We need to have a way to deal with this in our build tool chain permanently without humans as the failure point.
Not all breakages are equal. For example, something that break chroot creation should be avoided. Breaking critpath too ( rats-install and autoqa ).
Holding 1 library (let's say ogre, because it was updated not so long ago) because 1 consumer of the library is not updated is gonna annoy the maintainer of the library. And while someone could fix the packages, not all packagers are coders, not all upstream are reactive.
What about adding some timeout, or deciding that after a threshold of rebuilded packages, the packages are pushed from the tag ( provided this doesn't break chroot/critpath, of course )?
So we do not hold rawhide for too long, yet provides enough protection for users against breakage.
( of course, this approach is full of subtle issues, what if 2 libraries break at the same time, what if I upgrade my package while on of the library is broken, etc )
On Wed, 23 Jan 2013 20:02:44 +0000 "Daniel P. Berrange" berrange@redhat.com wrote:
If the deps provided by a new build have changed, then verify that the change deps don't cause breakage. If they do, then do not allow the new build into the rawhide target. Queue the build in some kind of build specific 'pending' target. Let all the broken deps be fixed in that pending target, and then atomically merge that target back into rawhide target. Basically at no time ever, should you allow broken deps to make it into rawhide. As long as our build process allows for broken deps, then rawhide is going to be a trainwreck of some kind or another. Requiring people to pre-announce deps breakage to let people promptly rebuild has unequivocably failed to solve the broken deps problems. We need to have a way to deal with this in our build tool chain permanently without humans as the failure point.
I agree. We could possibly use something like:
build finishes for f19, lands in a f19-pending autoqa runs on the package, if it passes, tag it in to f19 if it doesn't, mail maintainer/etc about it. If there's some compelling reason it has to land, the maintainer can override and tag into f19 directly. (and then explain themselves :)
But we would need the autoqa part to exist and be ready for this.
kevin
On Wed, Jan 23, 2013 at 02:23:01PM -0700, Kevin Fenzi wrote:
On Wed, 23 Jan 2013 20:02:44 +0000 "Daniel P. Berrange" berrange@redhat.com wrote:
If the deps provided by a new build have changed, then verify that the change deps don't cause breakage. If they do, then do not allow the new build into the rawhide target. Queue the build in some kind of build specific 'pending' target. Let all the broken deps be fixed in that pending target, and then atomically merge that target back into rawhide target. Basically at no time ever, should you allow broken deps to make it into rawhide. As long as our build process allows for broken deps, then rawhide is going to be a trainwreck of some kind or another. Requiring people to pre-announce deps breakage to let people promptly rebuild has unequivocably failed to solve the broken deps problems. We need to have a way to deal with this in our build tool chain permanently without humans as the failure point.
I agree. We could possibly use something like:
build finishes for f19, lands in a f19-pending autoqa runs on the package, if it passes, tag it in to f19 if it doesn't, mail maintainer/etc about it.
If it doesn't pass I think it is critical to move it into a private tag eg f19-<RPMNAME>-pending, so that you can do the downstream rebuilds against it in isolation from any other builds in f19-pending or rawhide.
If there's some compelling reason it has to land, the maintainer can override and tag into f19 directly. (and then explain themselves :)
But we would need the autoqa part to exist and be ready for this.
I dunno what the scope of autoqa is, but even a simple dependancy check would be sufficient - no need to wait for a full automated qa system to be built.
IMHO the key is that if we ever want rawhide to be generall useful and consumable, we need to fix the "broken deps" problem permanently. As long as we have the current mess of broken deps in rawhide, I'll never go anywhere near rawhide for my day to day work.
Daniel
On Thu, 2013-01-24 at 11:18 +0000, Daniel P. Berrange wrote:
But we would need the autoqa part to exist and be ready for this.
I dunno what the scope of autoqa is, but even a simple dependancy check would be sufficient - no need to wait for a full automated qa system to be built.
A 'simple' dependency check is one of the two main autoqa tests as things stand: depcheck. It is not sufficiently reliable to be enforced, and it cannot be made so without some serious engineering work, so tflink tells me. There's really no such thing as a 'simple' dependency check which will be at all reliable.
as depcheck stands, it's useful in an advisory capacity, but not accurate enough to be enforced.
Kevin Fenzi wrote:
build finishes for f19, lands in a f19-pending autoqa runs on the package, if it passes, tag it in to f19 if it doesn't, mail maintainer/etc about it. If there's some compelling reason it has to land, the maintainer can override and tag into f19 directly. (and then explain themselves :)
I'm sure the override would happen with every GCC upgrade. The soname of Libgnat changes with every GCC release, so all the Ada packages have to be rebuilt in dependency order. It's normally just a rebuild with no bugfixing, but since it has to be manually started and monitored it will always take some time. I imagine that quite a few people would be annoyed if GCC would be delayed while we rebuild the Ada packages.
Björn Persson
On Thu, Jan 24, 2013 at 03:54:33PM +0100, Björn Persson wrote:
Kevin Fenzi wrote:
build finishes for f19, lands in a f19-pending autoqa runs on the package, if it passes, tag it in to f19 if it doesn't, mail maintainer/etc about it. If there's some compelling reason it has to land, the maintainer can override and tag into f19 directly. (and then explain themselves :)
I'm sure the override would happen with every GCC upgrade. The soname of Libgnat changes with every GCC release, so all the Ada packages have to be rebuilt in dependency order. It's normally just a rebuild with no bugfixing, but since it has to be manually started and monitored it will always take some time. I imagine that quite a few people would be annoyed if GCC would be delayed while we rebuild the Ada packages.
IMHO this kind of focus is why rawhide is in the state it is today. Minimising work for package maintainers, at the cost of making rawhide unusable for people who ought to be able to test it. As long as we have that mindset rawhide will continue to eat babies and kill your ponies. Delaying a new package while we fix broken deps by rebuilding things that were broken is a *good* thing - provided we have a means to ensure the downstream rebuilds can be performed without being delayed by non-responsive maintainers.
Daniel
Daniel P. Berrange wrote:
Delaying a new package while we fix broken deps by rebuilding things that were broken is a *good* thing - provided we have a means to ensure the downstream rebuilds can be performed without being delayed by non-responsive maintainers.
In principle it should be possible to do the rebuild entirely automatically, if we allow automated processes to push to Git. Even a client-side tool to do rebuilds in dependency order would help, but I haven't yet figured out how to write one.
Björn Persson
On Wed, 23 Jan 2013 09:55:07 -0700 Kevin Fenzi kevin@scrye.com wrote:
Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
As a non-Dev, but running rawhide. Hence reply here.
yum.conf skip_broken=1 by default. It would remove some of the dep coughs.
Maybe yum-local the critpath stuff? yum pluginise the following: http://wwoods.fedorapeople.org/files/critical-path/old/critpath.py
modify yum-local to keep say 5 copies? (in case something hits the fan)
On Thu, 24 Jan 2013 15:06:50 +0000 Frank Murphy frankly3d@gmail.com wrote:
On Wed, 23 Jan 2013 09:55:07 -0700 Kevin Fenzi kevin@scrye.com wrote:
Greetings.
As some of you may know, I switched my laptop over to rawhide and have been posting a series of blogs about the various issues I have run into in the last month.
As a non-Dev, but running rawhide. Hence reply here.
Always good to have more people using rawhide, IMHO.
yum.conf skip_broken=1 by default. It would remove some of the dep coughs.
It could, but sometimes that causes issues like pulling in a bunch of 32bit packages to satisfy a dep.
Maybe yum-local the critpath stuff? yum pluginise the following: http://wwoods.fedorapeople.org/files/critical-path/old/critpath.py
I've been using yum-local here, but it's not as handy as I thought it might be. ;(
modify yum-local to keep say 5 copies? (in case something hits the fan)
Yeah, although you can always get them from koji.
kevin
Kevin Fenzi kevin@scrye.com a écrit:
modify yum-local to keep say 5 copies? (in case something hits the fan)
Yeah, although you can always get them from koji.
... provided your Rawhide system is in good enough shape to reach koji.
:)
On Thu, 24 Jan 2013 15:59:05 -0700 Kevin Fenzi kevin@scrye.com wrote:
http://wwoods.fedorapeople.org/files/critical-path/old/critpath.py
I've been using yum-local here, but it's not as handy as I thought it might be. ;(
modify yum-local to keep say 5 copies? (in case something hits the fan)
Yeah, although you can always get them from koji.
If on single rawhide machine, could be stuck with no network. keepcache=1 + http://freecode.com/projects/tidy-yum-cache-plugin Currently using it with: F16/17/18/F19-Rawhide
kevin