In the past, we would normally begin syncing soon to be EOL releases over to Archives so that the mirrors would be ready for it. We haven't done this for a bit and a couple of mirrors have complained that /pub/fedora is larger than they want to keep. I would like to get an official policy to them about how many releases that a mirror should expect to be in /pub/fedora and when we plan to archive off releases.
The steps for syncing/hardlinking releases was the following:
log into a box with read-write access to the the release tree. sudo su -s /bin/bash ftpsync cd /pub/fedora/linux/releases ls -l /pub/archive/fedora/linux/releases/ # to see if tree is already synced
If I were to archive off 23 I would now do the following:
cp -lr 23/ /pub/archive/fedora/linux/releases/
this should hardlink everything across so we save disk space.
cd ../updates cp -lr 23/ /pub/archive/fedora/linux/updates/ cd testings cp -lr 23/ /pub/archive/fedora/linux/updates/testing/
cd /pub/fedora-secondary/releases cp -lr 23/ /pub/archive/fedora-secondary/releases/ cd ../updates/ cp -lr 23/ /pub/archive/fedora-secondary/updates/ cd testing cp -lr 23/ /pub/archive/fedora-secondary/updates/testing/
Make any changes to scripts which do fullfilelist to make it know about new trees.
Announce the mirroring to mirrors-list and then wait 2 weeks. After two weeks are done, have mirrormanager point to the archive tree. Remove the data out of /pub/fedora and /pub/fedora-secondary Hardlink the file in a previous archived directory to the empty directory
Update the fullfilelist Email mirrorlist about the changes being completed
======
What should we tell the lists that the number of releases which will be in the main repository now? And how many releases should I archive off after F24 goes EOL next week?
On Thu, Aug 03, 2017 at 05:00:00PM -0400, Stephen John Smoogen wrote:
What should we tell the lists that the number of releases which will be in the main repository now? And how many releases should I archive off after F24 goes EOL next week?
Based on the mirror metadata, there's basically four tiers of traffic:
Substantial: F26, F25 (N, N-1) Moderate: F24, F23 (N-2, N-3) Still Chugging Along: F22, F21, F20 (N-4, N-5, N-6!) Marginal: Everything else. (Even F8.)
"Substantial" totals something like 150k unique IPs per day. "Moderate" totals ~ 60k. "Still Chugging" totals ~ 40k "Marginal" totals ~ 15k.
(For scale, all Fedora OS totals around 270k and all EPEL ~ 1.1m)
I think we already dropped F21 and earlier, right? I think F22 is pretty obvious. Long tail aside, our upgrade update has been pretty awesome recently, and I'm I'm leaning towards F23 as well, making the policy simply "N through N-2".
The mirrormanager stats tell a slightly different story, with updates-released-f21 getting hit like crazy -- I assume that's some sort of automated process (with it all coming from one or several IPs). Possibly the same thing is hitting F22. But if it's right that we already dropped F21, we seem to be surviving this just fine and adding F22 and F23 _probably_ won't hurt (right?).
On 3 August 2017 at 21:13, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Aug 03, 2017 at 05:00:00PM -0400, Stephen John Smoogen wrote:
What should we tell the lists that the number of releases which will be
in
the main repository now? And how many releases should I archive off after F24 goes EOL next week?
Based on the mirror metadata, there's basically four tiers of traffic:
Substantial: F26, F25 (N, N-1) Moderate: F24, F23 (N-2, N-3) Still Chugging Along: F22, F21, F20 (N-4, N-5, N-6!) Marginal: Everything else. (Even F8.)
"Substantial" totals something like 150k unique IPs per day. "Moderate" totals ~ 60k. "Still Chugging" totals ~ 40k "Marginal" totals ~ 15k.
(For scale, all Fedora OS totals around 270k and all EPEL ~ 1.1m)
I think we already dropped F21 and earlier, right? I think F22 is pretty obvious. Long tail aside, our upgrade update has been pretty awesome recently, and I'm I'm leaning towards F23 as well, making the policy simply "N through N-2".
The mirrormanager stats tell a slightly different story, with updates-released-f21 getting hit like crazy -- I assume that's some sort of automated process (with it all coming from one or several IPs). Possibly the same thing is hitting F22. But if it's right that we already dropped F21, we seem to be surviving this just fine and adding F22 and F23 _probably_ won't hurt (right?).
The main thing is that those people are just checking.. and aren't getting any updates so they are just as happy getting them from 8 servers or 100 servers. Yes some people may be asking for 'new' packages for those systems but they are pretty much already all there. I don't see much bonus for having the mirrors wanting to cover us drop because we are over 1 TB of data in our what we want people to mirror. Especially as our disk usage may go up with flatpacks and other image formats.
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists. fedoraproject.org
On Thu, Aug 03, 2017 at 09:24:40PM -0400, Stephen John Smoogen wrote:
I think we already dropped F21 and earlier, right? I think F22 is pretty obvious. Long tail aside, our upgrade update has been pretty awesome recently, and I'm I'm leaning towards F23 as well, making the policy simply "N through N-2".
[...]
The main thing is that those people are just checking.. and aren't getting any updates so they are just as happy getting them from 8 servers or 100 servers. Yes some people may be asking for 'new' packages for those systems but they are pretty much already all there. I don't see much bonus for having the mirrors wanting to cover us drop because we are over 1 TB of data in our what we want people to mirror. Especially as our disk usage may go up with flatpacks and other image formats.
Makes sense. The above proposal is pretty aggressive, basically keeping a release for ~6 months after EOL, which I think makes sense because we support upgrades from N-2. This way, updates will be available for users of F24 who want to update before upgrading to F26 can do that all the way through the F27 release.
I'm willing to be convinced that this isn't significant enough to even bother with, though, and that we should be even more aggressive and just archive all EOL releases.
On 08/03/2017 09:24 PM, Stephen John Smoogen wrote:
Especially as our disk usage may go up with flatpacks and other image formats.
Sort of unrelated, but I've been talking with Owen Taylor about getting flatpaks into our container registry. So far, our container registry isn't backed by the mirrors. I guess it *could* be backed by them in the future though.
On 08/04/2017 10:30 AM, Randy Barlow wrote:
On 08/03/2017 09:24 PM, Stephen John Smoogen wrote:
Especially as our disk usage may go up with flatpacks and other image formats.
Sort of unrelated, but I've been talking with Owen Taylor about getting flatpaks into our container registry. So far, our container registry isn't backed by the mirrors. I guess it *could* be backed by them in the future though.
this is a big reason why I want to move https://pagure.io/fedora-infrastructure/issue/6022 along. That is at least one very good performance mirror we can put containers onto.
That ticket has not had very much progress, though.
Dusty
On Fri, Aug 04, 2017 at 11:49:37AM -0400, Dusty Mabe wrote:
Sort of unrelated, but I've been talking with Owen Taylor about getting flatpaks into our container registry. So far, our container registry isn't backed by the mirrors. I guess it *could* be backed by them in the future though.
this is a big reason why I want to move https://pagure.io/fedora-infrastructure/issue/6022 along. That is at least one very good performance mirror we can put containers onto. That ticket has not had very much progress, though.
I'm tempted to poke more at that ticket, but I think priorities right now need to go towards the F27 stuff; better mirroring and better use of Amazon's generous support isn't preventing us from doing anything right now.
infrastructure@lists.fedoraproject.org