Hi all,
I run an established public mirror for EPEL and Rocky, and wanted to go ahead and add Fedora into it as well.
I am running the quick-fedora-script for this. Started it around 10am yesterday, checked on it today around 8am and it still had not even began downloading anything yet. Just spinning the CPU core it's using at 100%. The log file showed it still at "Receiving file list" stage.
22 hours, no progress. That's insane.
Is there a better supported way of getting the initial sync? Or is it really just wait it out and hope it finishes?
Dnia Fri, May 19, 2023 at 08:27:39AM -0500, Russell Jones napisał(a):
I am running the quick-fedora-script for this. Started it around 10am yesterday, checked on it today around 8am and it still had not even began downloading anything yet. Just spinning the CPU core it's using at 100%. The log file showed it still at "Receiving file list" stage. 22 hours, no progress. That's insane.
The script is not quite easy to run so I rather expect some configuration mistake, not such a slowness.
Is there a better supported way of getting the initial sync? Or is it really just wait it out and hope it finishes?
Just rsync it from elsewhere, e.g. from the source you use in the script.
R.
Thanks,
I considered trying just a bare rsync, but that's where it's stuck at is the rsync process.
I'll try just rsync with the flags I use for mirroring EPEL (I don't use the quick sync script for that repo), see if it makes a difference.
On Fri, May 19, 2023 at 8:41 AM Rafal Maszkowski rzm@icm.edu.pl wrote:
Dnia Fri, May 19, 2023 at 08:27:39AM -0500, Russell Jones napisał(a):
I am running the quick-fedora-script for this. Started it around 10am yesterday, checked on it today around 8am and it still had not even began downloading anything yet. Just spinning the CPU core it's using at 100%. The log file showed it still at "Receiving file list" stage. 22 hours, no progress. That's insane.
The script is not quite easy to run so I rather expect some configuration mistake, not such a slowness.
Is there a better supported way of getting the initial sync? Or is it really just wait it out and hope it finishes?
Just rsync it from elsewhere, e.g. from the source you use in the script.
R.
„Walczy on z całym zapamiętaniem przeciwko intelektowi” - z akt personalnych prof. A. Baeumlera
OK, yeah, it's already actually grabbing content now. Took all of 10 seconds.
Something's up with that quick sync script. I'll stick to normal rsync.
On Fri, May 19, 2023 at 8:48 AM Russell Jones arjones85@gmail.com wrote:
Thanks,
I considered trying just a bare rsync, but that's where it's stuck at is the rsync process.
I'll try just rsync with the flags I use for mirroring EPEL (I don't use the quick sync script for that repo), see if it makes a difference.
On Fri, May 19, 2023 at 8:41 AM Rafal Maszkowski rzm@icm.edu.pl wrote:
Dnia Fri, May 19, 2023 at 08:27:39AM -0500, Russell Jones napisał(a):
I am running the quick-fedora-script for this. Started it around 10am yesterday, checked on it today around 8am and it still had not even
began
downloading anything yet. Just spinning the CPU core it's using at 100%. The log file showed it still at "Receiving file list" stage. 22 hours, no progress. That's insane.
The script is not quite easy to run so I rather expect some configuration mistake, not such a slowness.
Is there a better supported way of getting the initial sync? Or is it really just wait it out and hope it finishes?
Just rsync it from elsewhere, e.g. from the source you use in the script.
R.
„Walczy on z całym zapamiętaniem przeciwko intelektowi” - z akt personalnych prof. A. Baeumlera
Dnia Fri, May 19, 2023 at 08:49:56AM -0500, Russell Jones napisał(a):
OK, yeah, it's already actually grabbing content now. Took all of 10 seconds. Something's up with that quick sync script. I'll stick to normal rsync. On Fri, May 19, 2023 at 8:48 AM Russell Jones arjones85@gmail.com wrote:
I considered trying just a bare rsync, but that's where it's stuck at is the rsync process. I'll try just rsync with the flags I use for mirroring EPEL (I don't use the quick sync script for that repo), see if it makes a difference.
Maybe it was problem with the rsync? Once I managed to configure the script it works for me quite well. It saves a lot of resouces and time. It fails when partition is filled, and in that case I recover by a plain rsync --delete ….
R.
On Fri, May 19, 2023 at 03:56:36PM +0200, Rafal Maszkowski wrote:
Dnia Fri, May 19, 2023 at 08:49:56AM -0500, Russell Jones napisał(a):
OK, yeah, it's already actually grabbing content now. Took all of 10 seconds. Something's up with that quick sync script. I'll stick to normal rsync. On Fri, May 19, 2023 at 8:48 AM Russell Jones arjones85@gmail.com wrote:
I considered trying just a bare rsync, but that's where it's stuck at is the rsync process. I'll try just rsync with the flags I use for mirroring EPEL (I don't use the quick sync script for that repo), see if it makes a difference.
Maybe it was problem with the rsync? Once I managed to configure the script it works for me quite well. It saves a lot of resouces and time. It fails when partition is filled, and in that case I recover by a plain rsync --delete ….
It really does save a lot of useless i/o on the master mirrors. ;)
That said, I ran into what sounds like the exact same issue here recently. I wasn't fully able to pin it down, but I think it might be a strange rsync bug. If the list of files to transfer is really large, things go slower and slower over time and it never gets to transfering.
What version of rsync are you using there? On the machine where I was seeing it, it was rhel9 with rsync-3.2.3-18.el9.x86_64 Updating it to the rsync version from fedora and it started working normally again, but I also did a full rsync, make sure time was correct, and a bunch of other things.
kevin
This is 3.1.3 on Rocky 8. Yes as I watched the strace output, it'd start out checking files pretty quick. When I checked it this morning it was checking maybe 1 file a second.
I'll try compiling the latest version of rsync after I get this initial sync done, and then running the quick sync script against my already-downloaded files with the new version of rsync. See if it makes a difference.
On Fri, May 19, 2023 at 11:35 AM Kevin Fenzi kevin@scrye.com wrote:
On Fri, May 19, 2023 at 03:56:36PM +0200, Rafal Maszkowski wrote:
Dnia Fri, May 19, 2023 at 08:49:56AM -0500, Russell Jones napisał(a):
OK, yeah, it's already actually grabbing content now. Took all of 10 seconds. Something's up with that quick sync script. I'll stick to normal rsync. On Fri, May 19, 2023 at 8:48 AM Russell Jones arjones85@gmail.com
wrote:
I considered trying just a bare rsync, but that's where it's stuck
at is
the rsync process. I'll try just rsync with the flags I use for mirroring EPEL (I don't
use
the quick sync script for that repo), see if it makes a difference.
Maybe it was problem with the rsync? Once I managed to configure the script it works for me quite well. It saves a lot of resouces and time. It fails when partition is filled, and in that case I recover by a plain rsync --delete ….
It really does save a lot of useless i/o on the master mirrors. ;)
That said, I ran into what sounds like the exact same issue here recently. I wasn't fully able to pin it down, but I think it might be a strange rsync bug. If the list of files to transfer is really large, things go slower and slower over time and it never gets to transfering.
What version of rsync are you using there? On the machine where I was seeing it, it was rhel9 with rsync-3.2.3-18.el9.x86_64 Updating it to the rsync version from fedora and it started working normally again, but I also did a full rsync, make sure time was correct, and a bunch of other things.
kevin _______________________________________________ Mirror-admin mailing list -- mirror-admin@lists.fedoraproject.org To unsubscribe send an email to mirror-admin-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/mirror-admin@lists.fedoraproje... Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
mirror-admin@lists.fedoraproject.org
-
Kevin Fenzi -
Rafal Maszkowski -
Russell Jones