mohanboddu reported a new issue against the project: `releng` that you are following: `` The fedora 27 schedule[1] has a mass rebuild scheduled for Jul 12, We need to plan and coordinate all tasks in preparation for it.
[1] https://fedoraproject.org/wiki/Releases/27/Schedule ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
kevin added a new comment to an issue you are following: `` Please note: https://pagure.io/fesco/issue/1735 before starting the mass rebuild.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
dgregor added a new comment to an issue you are following: `` Did this mass rebuild happen? I am looking into the scalability of the Fedora koji deployment and am curious how many packages were rebuilt and how long that took. Any details you have would be appreciated. Thanks. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
mohanboddu added a new comment to an issue you are following: `` @dgregor We are waiting on https://pagure.io/fesco/issue/1735 and once thats cleared I can inform you when we are going to start and we can get some statistics out of it. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
jkurik added a new comment to an issue you are following: `` @mohanboddu: what is the status of the mass rebuild ? https://pagure.io/fesco/issue/1735 does not seem to block it anymore. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
mohanboddu added a new comment to an issue you are following: `` @jkurik Mass rebuild started on Tue night US EDT time. Its still running, its going to take around 5-7 days. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` We may need another mass rebuild to revert a conceptual linking change in binutils: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/... ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
nickc added a new comment to an issue you are following: `` Please may we have another mass rebuild ?
I have updated the rawhide binutils to implement the change suggested by Florian, but in order for it to take effect all libraries and compiled executables need to be rebuilt. If it helps, the rebuild is only necessary for the PowerPC architecture, not any of the others.
Thanks. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
mohanboddu added a new comment to an issue you are following: `` @nickc We are planning on doing another mass rebuild skipping noarch packages. But it has to wait until the current mass rebuild is done. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
pfrields added a new comment to an issue you are following: `` Starting another mass rebuild immediately following the one we just had will affect a number of critical infrastructure team projects, such as the pagure-dist-git deployment. That is critical for a number of other people outside our teams. Can we schedule the mass rebuild to follow that? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` Rawhide is currently in a really bad shape on ppc64le because key libraries were mislinked. We'll have to recompile an unknown number of components manually, but it should be doable. This work is is required for the second mass rebuild anyway, otherwise there will be too many failures. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
kevin added a new comment to an issue you are following: `` If you could let us know here when those key packages are done we can figure out when to do the next mass rebuild.
I will note that the only interaction between the mass rebuild and the pagure over distgit work is committing the spec bump to distgit. Once those are done the pagure over distgit work could go ahead and the mass rebuild will just build everything its already building. Also the next mass rebuild only will need archfull packages which will cut down the time/number involved a great deal. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
pfrields added a new comment to an issue you are following: ``
If you could let us know here when those key packages are done we can figure out when to do the next mass rebuild. I will note that the only interaction between the mass rebuild and the pagure over distgit work is committing the spec bump to distgit. Once those are done the pagure over distgit work could go ahead and the mass rebuild will just build everything its already building. Also the next mass rebuild only will need archfull packages which will cut down the time/number involved a great deal.
Right -- that's how I understood the issue as well. The only concern I have is giving @pingou enough clearance to do his work. AIUI there have already been some delays and each additional one at this point makes it more unlikely we can hit milestones by Flock time. It doesn't seem like there's a good alternative where everyone gets things quickly, sorry for that. My hope is that @pingou, @mohanboddu, and other folks can find the least-bad solution. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` What's the repository used for the current ppc64le buildroot? I tried this:
* https://koji.fedoraproject.org/repos/rawhide/latest/ppc64le
But it's not what the builders use. For example, they install rpm-build 4.13.0.1-40, but the repository above currently has version 4.13.0.1-39 only.
I need the right repository to minimize rebuilds, otherwise I'm down to trial-and-error. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` I'm now using
* https://kojipkgs.fedoraproject.org/repos/f27-build/latest/ppc64le
for reference. It seems more recent than the `rawhide` repository, so maybe this is the right one.
There seems to be an ongoing issue with the buildroot generation, and new packages do not show up in it. Until that happens, I cannot start the targeted rebuild effort in earnest. (The immediate goal is to unbreak cmake.) ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
kevin added a new comment to an issue you are following: `` f27-build is correct. "rawhide" is just one we have as an alias.
While the mass rebuild was finishing it may have been a bottleneck with signing packages. I am not sure what it is currently, but will investigate. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
pingou added a new comment to an issue you are following: `` So what is the status on this? Do we start a new mass-rebuild? When? How long will it take? (time-wise what impacts me is how long it takes to commit and queue all the builds, not actually finish the builds)
I saw some discussion on the devel list about progress being made over the week-end but it wasn't really clear for me what the outcome was :s ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` My targeted rebuilds for the ppc64le issue are still ongoing, but are nearing completion. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
pingou added a new comment to an issue you are following: ``
My targeted rebuilds for the ppc64le issue are still ongoing, but are nearing completion.
So all interaction with git is done? There is no script running that would crash is git was not accessible for a minute or two? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` @pingou, it's mostly manual anyway. I don't need dist-git right now. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
fweimer added a new comment to an issue you are following: `` I believe the ppc64le cleanup is done for now (as long as there aren't any untagged ppc64le builds waiting to be tagged into f27-build at this point). ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
kevin added a new comment to an issue you are following: `` Just to keep this ticket updated:
* all the noarch packages that were rebuilt in the rebuild last week have been tagged into rawhide. (todays rawhide compose failed, but they should show up soon).
* All the archfull packages are being mass rebuilt again now in a new tag.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6898
rel-eng@lists.fedoraproject.org
-
Dennis Gregorovic -
Florian Weimer
-
Jan Kurik
-
Kevin Fenzi
-
Mohan Boddu
-
Nicholas Clifton
-
Paul W. Frields
-
Pierre-YvesChibon