On Fri, Jun 05, 2015 at 05:09:06PM +0200, Pierre-Yves Chibon wrote:
Good morning everyone,
Here is another release of the day, MirrorManager2: 0.2.0
And here is the changelog:
* Fri Jun 05 2015 Pierre-Yves Chibon <pingou(a)pingoured.fr> - 0.2.0-1
- Update to 0.2.0
- Include the background header file in MM2 itself (Adrian Reber)
- Support always update hosts which are unreachable in the crawler (Adrian
Reber)
- Adjust the spec file to the systemd packaging guidelines for Fedora
- Multiple improvements to the crawler, including a start of a canary mode
(Adrian Reber)
- Offer possibility to sort by product, bringing back MM1 behavior (Adrian
Reber)
- Couple of UI fixes about who is allowed to access what
- Fix peer ASNs (in the same spirit, who can access)
- Create noauthed master for mirror publiclist so that it can be cached in
memcachd (Patrick Uiterwijk)
I think this should read varnish and not memcachd. I think it still uses
the 'default' cache age of 120 seconds. The publiclist used to be
created once or twice a day so that we could easily keep the cache much
longer than the current 120 seconds. I think 12 hours would be no
problem, as this list is 'only' used as an overview and independent of
the results of the actual yum/dny mirrorlist/metalink.
- Fix the report_mirror to correctly catch the
xmlrpclib.ProtocolError
- Add a new utility script to upgrade repo from -alpha or -beta to release
- Adjust the logrotate configuration to fix the permission denied error
- Create 2 API endpoints, one for zodbot's .mirroradmin and one for nagios
Note: this is currently only in *staging*, we did not feel like pushing this to
prod on a Friday afternoon :)
We'll likely push on Monday though, assuming nothing burns down over the
week-end.
I tested the new canary and repodata crawl mode and I was able to crawl
all mirrors in a much short time:
$ time mm2_crawler --repodata -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real 36m38.639s
user 27m20.689s
sys 3m13.829s
$ time mm2_crawler --canary -c ~/mirrormanager2.cfg --threads 75 --timeout 30
real 6m33.624s
user 0m16.061s
sys 0m2.279s
Canary mode only checks if at least one base URL of all categories on
the mirror works. If no URL works the whole mirror is marked as not
being update. This also means that the mirror will not be re-enabled
until a full crawl or successful report_mirror run. This behaviour that
canary mode disables the mirror until a full crawl or report_mirror run
could be changed but would require database changes and therefore we
decided to support this form of canary mode as a first step.
Repodata mode crawls all repodata directories and only marks those
directories as being up to date or not. A mirror will not be completely
disabled in repodata mode in contrast to canary mode.
As canary mode seems to finish in under 10 minutes we can now think
about using full crawls and canary or repodata mode in combination to
get better overview about the availability and the status of the
mirrors.
Adrian