Hi,
As per the latest status report in the ticket [1], I'd like to move
forward with the next step.
I'd like to have all existing source files hardlinked to their new
path, from:
/$module/$source/$hash/$sources
To:
/$module/$source/md5/$hash/$sources
Doing this makes it easier to continue with the changes to fedpkg:
1. all files uploaded since June 5 exist both at the old and new paths
2. all files uploaded before only exist at the old path
3. fedpkg currently downloads from the old path, but we'll eventually
make it download from the new path
If we changed fedpkg right now, then it needs to try downloading at the
new path, and if that fails, fallback on the old path. I have code
doing just that ready to be sent as a pull request if needed.
However, if we first hardlink all existing source files to the new
path, then we don't even need the fallback in fedpkg, which makes the
code much simpler.
Note: hardlinking to the sha512 path can come at a later stage, for
now I am only talking about hardlinking to the new md5 path.
Attached is an Ansible patch that adds a script I wrote, which iterates
over all source files in the lookaside cache, and:
- if the file is at the old path, verify its md5 and hardlink it to
the new path
- if the file is at the new path, just verify its md5
I ran that script in staging with a limited subset of the lookaside
cache [2] and it did the job.
I'd like some review of the script first, and then someone to run it in
production. The script won't do anything without the --perform option,
so it's safe to try it out first:
$ make-new-lookaside-links /srv/cache/lookaside/pkgs/
At that point I'd like a chance to review the output, to make sure
everything is going as expected. :)
When ready to actually do it: [3]
$ make-new-lookaside-links --perform /srv/cache/lookaside/pkgs/
I'm happy to discuss this more here on on IRC, and I'll try my best to
be available when someone runs it in production.
Thanks,
[1]
https://fedorahosted.org/rel-eng/ticket/5846#comment:23
[2] all the 'ad*' packages, thanks to Ralph for syncing them over from
prod
[3] running the script with the --perform option requires doing so as a
user with write permission inside of /srv/cache/lookaside/pkgs/...
Either apache or root should work.
--
Mathieu