jlebon reported a new issue against the project: `releng` that you are following: `` We have internal testers which essentially check for new commits by polling the `fedora-atomic/25/x86_64/updates/docker-host` ref on https://kojipkgs.fedoraproject.org/atomic/25. When the ref changes, a new test is triggered. We've been seeing failures in which the ref gets updated, but the commit object to which the ref points to is not yet in the repo.
We need to make sure that all the refs are being updated safely: first objects, then refs. There is a more complete [rsync-repos](https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos) script that knows how to safely do this.
For the updates ref in specific, judging from the masher code, it seems like we're operating on the live repo and then "waiting" for it sync out?
https://github.com/fedora-infra/bodhi/blob/1a2871520d719c44f5cab37b2a5554741...
https://github.com/fedora-infra/bodhi/blob/1a2871520d719c44f5cab37b2a5554741...
What kind of lag are we talking about? 10m? 30m? Is there any way to know when *all* the files have been synced out?
We might need to change this so that we compose on a *copy* of the repo, and then e.g. use the rsync script above to update the real repo. But this presumes that the order in which we rsync to the mount reflects the order in which it hits the public, which may not be true it seems? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
jlebon added a new comment to an issue you are following: `` Maybe an easy way to do this is to use `--write-commitid-to=` so that the compose doesn't actually update the ref, and then only update the ref once we hit mashtask.sync.done? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
jlebon added a new comment to an issue you are following: `` OK, I was able to do some measurements. From the time the ref is updated to the time the object is accessible is around 30 minutes. That's a *massive* window which will realistically affect not just machines that continuously poll, but users following the updates stream as well. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
kevin added a new comment to an issue you are following: `` https://infrastructure.fedoraproject.org/cgit/ansible.git/tree/roles/bodhi2/... is the script and where it syncs. This script runs every 30min (at :15 and :45).
But I don't understand why there would be a 30min delay. The script should have everything synced after it runs and we copy objects first and then refs.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
kevin added a new comment to an issue you are following: `` Oh, of course bodhi could be syncing into place while the above cron is trying to sync out to master mirrors, resulting in partial doom.
We can/should replace the cron with a fedmsg driven trigger for syncing that runs only when bodhi is done with something and ready for it to sync out. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
jlebon added a new comment to an issue you are following: `` Right, we had a meeting about this with @puiterwijk a week or two ago, and I think in a nutshell what we discovered was that it was most likely because the cron takes time to sync the objects, and by the time it gets to rsync'ing the ref, another compose might have run which updated it.
We can/should replace the cron with a fedmsg driven trigger for syncing that runs only when bodhi is done with something and ready for it to sync out.
Yeah, that makes sense. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
parasense added a new comment to an issue you are following: `` Here is how I implement ostree rsync's to properly fence the refs.
rsync -rlpt --include=/objects --include=/objects/** --include=/deltas --include=/deltas/** --exclude=* SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --ignore-existing rsync -rlptvPH --include=/refs --include=/refs/** --include=/summary --include=/summary.sig --exclude=* SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --delete rsync -rlptvPH --include=/objects --include=/objects/** --include=/deltas --include=/deltas/** --exclude=* SOURCE/ostree/repo/ DESTINATION/ostree/repo/ --ignore-existing --delete
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
walters added a new comment to an issue you are following: `` https://github.com/ostreedev/ostree-releng-scripts/blob/master/rsync-repos is an implementation of this - I'd really like if we made use of ostree-releng-scripts rather than duplicating it. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
mohanboddu added a new comment to an issue you are following: `` Someone from infra or releng has to replace the cron job with a fedmsg driven job. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/6645
The issue: `Make sure OSTree repos are sync'ed correctly after the masher completes` of project: `releng` has been assigned to `mohanboddu` by ausil.
rel-eng@lists.fedoraproject.org