Fedora,
This is a reminder that the glibc team will be rebasing glibc in F21 to match glibc 2.20.
The plan remains largely as was written here: http://fedoraproject.org/wiki/Changes/GLIBC220
Only glibc 2.20 has ABI guarantees, and therefore we will move to 2.20 before F21 goes to GA to ensure that F21 has a strong ABI guarantee.
I expect this to be the process forever going forward:
* Rawhide tracks glibc master. * Fedora release is branched from Rawhide. * glibc release is made upstream. * Fedora branch is rebased on glibc upstream release to include ABI guarantees. * Fedora release goes GA.
The changes in the rebase should be minor since we've been tracking master the whole time.
Cheers, Carlos.
On Wed, Jul 30, 2014 at 01:18:57AM -0400, Carlos O'Donell wrote:
I expect this to be the process forever going forward:
- Rawhide tracks glibc master.
- Fedora release is branched from Rawhide.
- glibc release is made upstream.
- Fedora branch is rebased on glibc upstream release to include ABI guarantees.
- Fedora release goes GA.
The changes in the rebase should be minor since we've been tracking master the whole time.
Looks reasonable. Thanks for the update.
Hi
On Wed, Jul 30, 2014 at 1:18 AM, Carlos O'Donell wrote:
I expect this to be the process forever going forward:
- Rawhide tracks glibc master.
- Fedora release is branched from Rawhide.
- glibc release is made upstream.
- Fedora branch is rebased on glibc upstream release to include ABI guarantees.
- Fedora release goes GA.
The changes in the rebase should be minor since we've been tracking master the whole time.
I remember a similar process causing problems in Rawhide earlier and Jared Smith talking with the glibc team to ensure that only regular releases hit rawhide.
Rahul
On Wed, Jul 30, 2014 at 03:16:30PM -0400, Rahul Sundaram wrote:
- Rawhide tracks glibc master.
- Fedora release is branched from Rawhide.
- glibc release is made upstream.
- Fedora branch is rebased on glibc upstream release to include ABI guarantees.
- Fedora release goes GA.
The changes in the rebase should be minor since we've been tracking master the whole time.
I remember a similar process causing problems in Rawhide earlier and Jared Smith talking with the glibc team to ensure that only regular releases hit rawhide.
It would be nice if we're careful in staging ABI changes in the glibc master into Rawhide so it doesn't break. (Possibly something we can do with Taskotron?)
On Wed, Jul 30, 2014 at 03:36:34PM -0400, Matthew Miller wrote:
The changes in the rebase should be minor since we've been tracking master the whole time.
I remember a similar process causing problems in Rawhide earlier and Jared Smith talking with the glibc team to ensure that only regular releases hit rawhide.
The problem with that approach is that lots of bugs go unnoticed until very late in rawhide, resulting in those bugs being caught and fixed only post-release. Besides, this process is different in principle from the earlier one because the earlier process simply cut the latest master into a Fedora release just days before Fedora went gold.
In the current process the glibc release Fedora gets is decided well in advance, that is when the Fedora release branch is cut from rawhide. Combine that with the fact that glibc upstream freezes weeks before it releases and you'll see that the glibc release that makes it into a Fedora release is eventually well tested in rawhide *and* is not ancient.
It would be nice if we're careful in staging ABI changes in the glibc master into Rawhide so it doesn't break. (Possibly something we can do with Taskotron?)
ABI changes are a minor problem with respect to glibc because as a principle, we don't break ABI. That is, any ABI breakages that happen are usually bugs. We try to ensure that with a basic ABI checker in the glibc testsuite and in future we'll be looking to enhance our ABI checking capabilities. Taskotron (based on a quick read from the wiki) seems to be a good framework for this.
The bigger problem is bugs that brick your system because unlike kernel, there is currently no way to simply reboot into an older known-to-work glibc. We had one such situation last week and IIRC the last such situation before that was close to two years ago, so while it is serious, it is not frequent.
That said, we're also thinking of ways to recover from such botched updates.
Siddhesh
Hi
On Wed, Jul 30, 2014 at 11:12 PM, Siddhesh Poyarekar wrote:
The problem with that approach is that lots of bugs go unnoticed until very late in rawhide, resulting in those bugs being caught and fixed only post-release. ABI. That is, any ABI breakages that happen are usually bugs. We try to ensure that with a basic ABI checker in the glibc testsuite and in future we'll be looking to enhance our ABI checking capabilities. Taskotron (based on a quick read from the wiki) seems to be a good framework for this.
Yes. My concern is that glibc is using Rawhide as their continuous integration sandbox to shake out bugs as opposed to doing it elsewhere and just taking care of integration of releases when they are ready. If this viewed as a gap that needs to be fixed, that's fine for now.
Rahul
On Thu, Jul 31, 2014 at 09:03:45AM -0400, Rahul Sundaram wrote:
Hi
On Wed, Jul 30, 2014 at 11:12 PM, Siddhesh Poyarekar wrote:
The problem with that approach is that lots of bugs go unnoticed until very late in rawhide, resulting in those bugs being caught and fixed only post-release. ABI. That is, any ABI breakages that happen are usually bugs. We try to ensure that with a basic ABI checker in the glibc testsuite and in future we'll be looking to enhance our ABI checking capabilities. Taskotron (based on a quick read from the wiki) seems to be a good framework for this.
Yes. My concern is that glibc is using Rawhide as their continuous integration sandbox to shake out bugs as opposed to doing it elsewhere and just taking care of integration of releases when they are ready.
I'm actually not concerned about Rawhide being used like this. That's kind of the purpose of Rawhide.
However the previous problem(s -- multiple) was glibc using non-Rawhide for integration testing, especially just while we were trying to stablise Fedora for a release:
https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html
But 2.20 is a stable branch, right? So this shouldn't be a concern now.
Rich.
On Thu, Jul 31, 2014 at 03:06:00PM +0100, Richard W.M. Jones wrote:
However the previous problem(s -- multiple) was glibc using non-Rawhide for integration testing, especially just while we were trying to stablise Fedora for a release:
https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html
We're absolutely not going to rebase master in non-rawhide. Our intention is to get glibc 2.20 release out before Fedora turns beta.
But 2.20 is a stable branch, right? So this shouldn't be a concern now.
Yes, it is in what we call, a slushy freeze right now. Changes going in are mostly regression fixes and other fixes that should not have any generated code impact. There is however one change that the s/390 folks will find painful - s/390 broke ABI in 2.19 without a proper justification and that is now going to be reverted in 2.20 (and in 2.19, but we don't really care about 2.19 right now).
Siddhesh
On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar siddhesh@redhat.com wrote:
On Thu, Jul 31, 2014 at 03:06:00PM +0100, Richard W.M. Jones wrote:
However the previous problem(s -- multiple) was glibc using non-Rawhide for integration testing, especially just while we were trying to stablise Fedora for a release:
https://lists.fedoraproject.org/pipermail/devel/2011-October/158440.html https://lists.fedoraproject.org/pipermail/devel/2011-October/158662.html
We're absolutely not going to rebase master in non-rawhide. Our intention is to get glibc 2.20 release out before Fedora turns beta.
But 2.20 is a stable branch, right? So this shouldn't be a concern now.
Yes, it is in what we call, a slushy freeze right now. Changes going in are mostly regression fixes and other fixes that should not have any generated code impact. There is however one change that the s/390 folks will find painful - s/390 broke ABI in 2.19 without a proper justification and that is now going to be reverted in 2.20 (and in 2.19, but we don't really care about 2.19 right now).
To expand on this just a bit, the s/390 ABI break was done by the s/390 glibc port maintainers and was done intentionally. This wasn't a "oops, we accidentally broke the ABI" bug. It was something the s/390 maintainers decided was of minimal concern and it turns out they were wrong. In my experience, glibc goes above and beyond maintaining their software. This s/390 incident is an anomaly in an otherwise very controlled and well-run software project.
josh
On Thu, Jul 31, 2014 at 01:41:25PM -0400, Josh Boyer wrote:
On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar
There is however one change that the s/390 folks will find painful - s/390 broke ABI in 2.19 without a proper justification and that is now going to be reverted in 2.20 (and in 2.19, but we don't really care about 2.19 right now).
To expand on this just a bit, the s/390 ABI break was done by the s/390 glibc port maintainers and was done intentionally. This wasn't a "oops, we accidentally broke the ABI" bug. It was something the s/390 maintainers decided was of minimal concern and it turns out they were wrong. In my experience, glibc goes above and beyond maintaining their software. This s/390 incident is an anomaly in an otherwise very controlled and well-run software project.
(FYI) LWN article on this subject:
https://lwn.net/Articles/605607/
Rich.
On Thu, 31 Jul 2014 19:19:49 +0100 "Richard W.M. Jones" rjones@redhat.com wrote:
On Thu, Jul 31, 2014 at 01:41:25PM -0400, Josh Boyer wrote:
On Thu, Jul 31, 2014 at 12:34 PM, Siddhesh Poyarekar
There is however one change that the s/390 folks will find painful - s/390 broke ABI in 2.19 without a proper justification and that is now going to be reverted in 2.20 (and in 2.19, but we don't really care about 2.19 right now).
To expand on this just a bit, the s/390 ABI break was done by the s/390 glibc port maintainers and was done intentionally. This wasn't a "oops, we accidentally broke the ABI" bug. It was something the s/390 maintainers decided was of minimal concern and it turns out they were wrong. In my experience, glibc goes above and beyond maintaining their software. This s/390 incident is an anomaly in an otherwise very controlled and well-run software project.
(FYI) LWN article on this subject:
yeah, it can happen, the hardest thing was to come up with a solution that would allow to follow the regular Rawhide build process using koji-shadow.
Dan
On 07/31/2014 07:41 PM, Josh Boyer wrote:
To expand on this just a bit, the s/390 ABI break was done by the s/390 glibc port maintainers and was done intentionally. This wasn't a "oops, we accidentally broke the ABI" bug. It was something the s/390 maintainers decided was of minimal concern and it turns out they were wrong.
And we couldn't have avoided that if we used only released glibc versions in rawhide because the 2.19 release had the ABI change, too. It wasn't just in the git master branch.
On Thu, Jul 31, 2014 at 09:03:45AM -0400, Rahul Sundaram wrote:
Yes. My concern is that glibc is using Rawhide as their continuous integration sandbox to shake out bugs as opposed to doing it elsewhere and just taking care of integration of releases when they are ready. If this viewed as a gap that needs to be fixed, that's fine for now.
CI is indeed a gap that needs to be fixed, but rebasing in rawhide isn't nearly as fatal as it seems. The upstream glibc commit policy is quite stringent in that every commit is required to pass an extensive (but obviously not exhaustive) testsuite and as a result it is usually safe (as safe one would consider rawhide) to rebase from the latest commit. I've been doing regular rebases for a while now (almost a year AFAICT) and IIRC we have had just one serious problem last week where i686 boxes got bricked. In the end it was found to be a gcc bug that miscompiled some code in glibc.
That said, we still need a nicer way to back out from a broken glibc update into the previous known-to-work glibc similar to how the kernel does it and not need a rescue disk for it.
Siddhesh
On Thu, Jul 31, 2014 at 23:05:21 +0530, Siddhesh Poyarekar siddhesh@redhat.com wrote:
the latest commit. I've been doing regular rebases for a while now (almost a year AFAICT) and IIRC we have had just one serious problem last week where i686 boxes got bricked. In the end it was found to be a gcc bug that miscompiled some code in glibc.
If you figured out what was going on in time you could even recover with a yum downgrade in you had a root shell open (which I was able to do). So not everyone using i686 got their machines bricked.
On Wed, Jul 30, 2014 at 8:36 PM, Matthew Miller mattdm@fedoraproject.org wrote:
On Wed, Jul 30, 2014 at 03:16:30PM -0400, Rahul Sundaram wrote:
- Rawhide tracks glibc master.
- Fedora release is branched from Rawhide.
- glibc release is made upstream.
- Fedora branch is rebased on glibc upstream release to include ABI guarantees.
- Fedora release goes GA.
The changes in the rebase should be minor since we've been tracking master the whole time.
I remember a similar process causing problems in Rawhide earlier and Jared Smith talking with the glibc team to ensure that only regular releases hit rawhide.
It would be nice if we're careful in staging ABI changes in the glibc master into Rawhide so it doesn't break. (Possibly something we can do with Taskotron?)
Ultimately glibc as a core library should be following what the rest of the packages / features in development change process are required to do IE anything that might change the ABI should have landed before the mass rebuild so it can be addressed as part of that and hence be in a stable state at the same time as the other features are due to be complete.
The fact that a core library that's stability is critical to the distribution as a whole doesn't bother to adhere to this and while having gone through the hoops of putting in a feature change basically then proceeded to completely ignore the requirements of said process (ie they dumped a bunch of stuff on the wiki and haven't bothered to revisit it and update it since) is some what pathetic in my mind.
What happens if between now and when 2.20 goes GA there's a requirement of an ABI break and we need to rebuild a chuck of the distribution? It doesn't affect them as they don't need to do the work to make that happen.
Peter
On Thu, Jul 31, 2014 at 02:13:44PM +0100, Peter Robinson wrote:
The fact that a core library that's stability is critical to the distribution as a whole doesn't bother to adhere to this and while having gone through the hoops of putting in a feature change basically then proceeded to completely ignore the requirements of said process (ie they dumped a bunch of stuff on the wiki and haven't bothered to revisit it and update it since) is some what pathetic in my mind.
As an upstream policy, glibc does not break ABI[1] and we've tried to adhere to this ever since I started contributing to glibc (which is ~2 years ago). Any ABI break that happens is a bug and the ABI impact Carlos talks about is mainly due to such bugs being discovered in rawhide, which again is not very often.
Siddhesh
[1] We did unknowingly break ABI in S/390 in 2.19 and that will likely be fixed in 2.20 and require at least a partial rebuild (perl and its modules), but you would have needed the mass rebuild even if we would have directly introduced a 2.20 release.
-
Bruno Wolff III -
Carlos O'Donell -
Dan Horák -
Florian Weimer -
Josh Boyer -
Matthew Miller -
Peter Robinson -
Rahul Sundaram -
Richard W.M. Jones -
Siddhesh Poyarekar