On Fri, Jun 26, 2015 at 11:04:35AM -0600, Stephen John Smoogen wrote:
On 26 June 2015 at 10:11, Pierre-Yves Chibon
<pingou(a)pingoured.fr> wrote:
> So the UMDL script currently supports three ways of crawling the tree:
> * file
> * rsync
> * directory
>
> We, in Fedora, are only using the last one. I believe the `rsync` mode was added
> to support Ubuntu and the file mode is basically a simplified version of the
> directory mode, but that we do not use at at the moment.
>
> I would like to propose that we drop support for rsync. I feel that it may be
> simpler and easier to create an UMDL and a crawler for each distro that would
> like to use MirrorManager than maintaining a one-script-fits-all UMDL that is
> in fact tested for only one of the scenario.
> That being said, if we ever have interest from Ubuntu, CentOS or any other
> communities, we should definitively look into making the UMDL and crawler as
> re-usable as possible for them, but keeping the distro-specific bits separated.
>
I thought rsync was added for speed versus Ubuntu and that we actually
use it a lot on the crawlers... because that is the reason we needed
to move the memory usage up.. it uses a lot of memory but takes a lot
shorter period to 'crawl' or am I conflating two different things?
That's the rsync of the crawler scanning the mirrors, that's much faster
compared to http/ftp. umdl analyzing the master mirror can either use
the local filesystem or rsync or a file as input.
Adrian