Heya,
We're adding some more storage to our mirrors - how big is buffet?
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
Heya,
We're adding some more storage to our mirrors - how big is buffet?
Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to require mirroring fedora-buffet for new tier 1 mirrors sounds not so good anymore as that is really a lot.
Adrian
On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
Heya,
We're adding some more storage to our mirrors - how big is buffet?
Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to require mirroring fedora-buffet for new tier 1 mirrors sounds not so good anymore as that is really a lot.
It is a lot, but I think it should still be required. Tier1 should have everything any Tier2 might want to pull a subset of.
On Wed, 29 Apr 2020 at 20:22, Chuck Anderson cra@wpi.edu wrote:
On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
Heya,
We're adding some more storage to our mirrors - how big is buffet?
Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to require mirroring fedora-buffet for new tier 1 mirrors sounds not so good anymore as that is really a lot.
It is a lot, but I think it should still be required. Tier1 should have everything any Tier2 might want to pull a subset of.
In the past I would agree, but we (aka Fedora) had to drop mirroring archives from 2 of our systems (download-ib01/download-i2 and download-cc-rdu01) because we didn't have the disk space on these servers to do that much storage. I am not sure trying to shove every use case for a top mirror into one box is going to work anymore.
At the moment I only see the following mirrors offering archives via mirrormanager:
mirror.math.princeton.edu pubmirror1.math.uh.edu d2lzkl7pfhq30w.cloudfront.net # a fedora project caching server dl.fedoraproject.org
I expect there are probably private ones which I am not seeing via my simple curl |& grep but the majority of users who are grabbing stuff from archived data are only getting it from those 4 places. While it would be good for more, I can also see that the majority of mirror traffic is EPEL driven with only the latest 3 Fedora's taking up the majority of users. Having those fed fast is probably more important than Fedora 18.
Would a different tiering system work better in the same way as we have different rsync targets?
On Thu, Apr 30, 2020 at 05:27:18AM -0400, Stephen John Smoogen wrote:
On Wed, 29 Apr 2020 at 20:22, Chuck Anderson cra@wpi.edu wrote:
On Wed, Apr 29, 2020 at 10:50:00PM +0200, Adrian Reber wrote:
On Wed, Apr 29, 2020 at 04:44:08PM -0400, James Cassell wrote:
On Wed, Apr 29, 2020, at 1:31 PM, Theo Morra wrote:
Heya,
We're adding some more storage to our mirrors - how big is buffet?
Someone said 3.8 TB the other day.
Unfortunately I was off by one. It is 38TB. Knowing that, my proposal to require mirroring fedora-buffet for new tier 1 mirrors sounds not so good anymore as that is really a lot.
It is a lot, but I think it should still be required. Tier1 should have everything any Tier2 might want to pull a subset of.
In the past I would agree, but we (aka Fedora) had to drop mirroring archives from 2 of our systems (download-ib01/download-i2 and download-cc-rdu01) because we didn't have the disk space on these servers to do that much storage. I am not sure trying to shove every use case for a top mirror into one box is going to work anymore.
At the moment I only see the following mirrors offering archives via mirrormanager:
mirror.math.princeton.edu pubmirror1.math.uh.edu d2lzkl7pfhq30w.cloudfront.net # a fedora project caching server dl.fedoraproject.org
I see a couple more from my location:
ftp-stud.hs-esslingen.de fedora-archive.ip-connect.info mirrors.dotsrc.org fedora-archive.ip-connect.vn.ua fedora-mirror02.rbc.ru
I asked MirrorManager for Fedora 26 based mirrors, so this does not mean that all these mirrors mirror the complete archive.
I expect there are probably private ones which I am not seeing via my simple curl |& grep but the majority of users who are grabbing stuff from archived data are only getting it from those 4 places. While it would be good for more, I can also see that the majority of mirror traffic is EPEL driven with only the latest 3 Fedora's taking up the majority of users. Having those fed fast is probably more important than Fedora 18.
Would a different tiering system work better in the same way as we have different rsync targets?
What I would love to see in any tiering system we set up is full support for quick-fedora-mirror. If quick-fedora-mirror could handle a partial tree (not only via fedora-buffet) that would be perfect.
Adrian
mirror-admin@lists.fedoraproject.org
-
Adrian Reber -
Chuck Anderson -
James Cassell -
Stephen John Smoogen -
Theo Morra