On Fri, Nov 24, 2006 at 11:09:58AM +0100, Michael Schwendt wrote:
[...]
repoview
--------
Running it for all of Fedora Extras takes a lot of time. So long, that
even if you let the push script do its work in a background terminal, it
is still painful to see how long it takes to complete.
For unknown reasons we also create repoview pages for the "debug"
repositories. If it were just my own decision, I would stop doing that,
since I doubt those web pages are popular enough. Who really browses
repoview for debuginfo packages? We should expect debuginfo packages to be
available for every relevant package in the repository.
Yeah, repoview for debuginfo packages seems unnecessary. We don't even run
repoview for updates{,-testing} at the moment, but we can easily integrate
it with the new system if we want.
createrepo < 0.4.5 is unable to handle "unknown" files
in its repodata
directory. Therefore it conflicts with repoview and runs into a fatal
error condition with a premature program termination, leaving behind a
temporary ".olddir". This is especially ugly, since an administrator would
need to recover from that manually and either move back files from
".olddir" or delete it. But when deleted, the repoview tree is lost and is
created from scratch. I've been told that this is a problem for mirrors,
where the several thousands of files are re-examined for changes just
because of the fresh time stamps. So, as of a few weeks ago, the push
script works around that successfully with a repodata backup strategy
outside of createrepo.
The old updates system code (and the fedora-updates-clean cronjob) has hacks
around this, as well, when dealing with the updateinfo.xml.gz. I
haven't checked, but how does createrepo >= 0.4.5 deal with unknown
files now?
For createrepo and repoview to be run in the background as a
scheduled
job, it needs a local lock on the repository. Most likely _not_
fine-grained locking on every arch-specific sub-repository, because every
locking comes at a cost (especially when there are multiple jobs of
different priority waiting).
Yeah, there definitely needs to be mutual exclusion with the updates
staging repository.
repoclosure
-----------
Running this takes even more time. Currently, it examine all of Fedora
Extras + Core + Updates + Legacy in a background job after packages have
been published. The time it takes is approximately the difference between
the time stamps of the build report and the broken deps report.
I'd like to run repoclosure on the updates before pushing them out at
all (sure we have updates-testing, but I don't want that becoming
another rawhide).
I spoke with skvidal over the summer speeding up the
closure process, and seem to remember mention of caching the
provides/requires to gain some speed. We'll definitely have to look
into this, as this process is indeed dreadfully slow.
> ''' Pushing '''
> * Moves packages to proper updates stage
More "stages" which are understood by plague would be good. We only use
one stage, needsign, which is the build-results repository known to the
build servers.
Well, once we get the Brew situation figured out, we can use its
detached gpg signatures to help with the push process. Whatever stages
we want (pending/testing/needsign/pushed) should probably be implemented
in the updates system itself, to allow the buildsystem to only worry about
spitting out builds. Althought, my Brew-fu is weak, so we'll have to
see what it offers once/if it is released.
[..]
> * updates repo cleaner
> * remove old packages
So far, the script I named repoprune is much faster than repomanage and
simplifies Fedora Extras repository maintenance a lot, since it gets rid
of orphaned and out-of-date sub-packages automatically, too.
Awesome, I will see what I can do about integrating this tool into the
new system.
luke