On Thu, Apr 23, 2020 at 01:36:48PM -0700, Kevin Fenzi wrote:
On Thu, Apr 23, 2020 at 11:46:25AM +0200, Pierre-Yves Chibon wrote:
> Good Morning Everyone,
> Kevin and I had a discussion on IRC the other day about mirroring the ansible
> The options we considered are:
> For a first step I went with a third approach: a small python service that
> runs every 3 minutes (configurable): git fetch && git fsck (to ensure the
> is in a correct state). It actually doesn't run every 3 minutes, it calls
> then wait for 3 minutes then calls git again and so on. So if git was ever to
> takes 4 minutes to run, there is no risk of the program stepping on its own
I'm wondering if that might be overkill.
Really if pagure.io blows up, we want to be able to look at the repo we
have on batcave01 and be able to still operate while we recover
pagure.io. If someone pushed or merged some commits in between the last
time we synced and blowup, someone should have them in their local repo
right? so we could just push them to the repo on batcave and be back
completly in sync, no?
Someone will have the commits sure, but since it's a PR, you could be merging a
couple of commits from me while I am asleep. If pagure goes down right after the
merge, you will not have the commits on your fork and batcave may not have
synced them, leading to these commits not being accessible.
This isn't a situation we couldn't recover from, but one of the two repos will
need to get a force-push once they are both available again.
but I guess it doesn't hurt to do every 3minutes, just some BW.
I think the fedora-messaging client would reduce the BW usage, and I think I
have it ready, so we may be able to go with it from the start.
Another option I mentioned the other day when we were talking, but I
didnt really explain well was that we could wrap ansible-playbook into a
wrapper that runs a git pull then ansible-playbook... so it would always
be in sync at first you run it. That could be messy tho if there's
another process doing pulls, etc. So I guess scrap that idea.
> I would like to propose that install and configure this for a test.
> I propose that we keep /git/ansible.git as is and just have a
> /git/mirrors/ansible.git which will be the mirror from pagure.
> We can migrate the hook from /git/ansible.git to /git/mirrors/ansible.git so
> /srv/web/infra/ansible (ie: the exploded tree) is coming from the mirror.
I think that could be confusing. If we switch we should switch and make
all the old checkouts no longer work for pushing, or we are going to
cause a lot of confusion. I'd say move the old one to some .back name
and chmod 000 it.
Works for me, I was thinking to keep it as some sort of backup that would be
updated hourly or so and thus would give us a little breathing room in case of
disaster (quickly diagnosed).
I'm ok with your approach that the breathing room gained may be overkilled there
(the chances that we diagnose an issue and think about stopping the service
while we're trying to debug the issue and that the service hasn't synced in the
mean time seems... low).
Also, there is the question about the existing git repo for
fedora-infrastructure on pagure. It has stuff in it now. Do we
completely replace it with the current ansible repo? Or do we stream all
the ansible commits on top of it so current checkouts would just be able
to pull? Or do we put it on another project. I'd really prefer to have
it on the same project our tickets are to avoid confusion. I kinda think
it might be nice to stream it on top of the existing repo so it's not
broken, but not sure how long or messy that would be.
I was going for: https://pagure.io/Fedora-Infra/ansible/
which is marked as
being the future location of our ansible repo and already has a copy of it.
I had not realized we have content in https://pagure.io/fedora-infrastructure/
If we really want to have the ansible repo in that repo, the easiest may be to
rebase these commits on the top of ansible and force push to that repo (so
current ansible clone would pull fine once they've updated their urls=).
> I guess the lag time to get /git/mirrors updated will quickly be
annoying (up to
> 3 minutes between when the PR is merged and when the playbook is updated), so
> if we like the workflow we will likely want to switch pretty quickly to a
> message-based service and then we could keep the current cron-like service to
> run hourly and update /git/ansible.git which would then become some kind of
So, we already have fedmsg-hub listening on batcave01 for fas messages
(which no longer happen). We could just reuse that? Or a
fedora-messaging version of it I guess would be better.
Almost there :)
> What do you think?
> FBR worthy? (ie: if F32 goes out next week, I think we can wait, if it doesn't
> do we want to try this sooner?)
Wait until after freeze. We have waited this long and we still have
release things to do in ansible. I don't want us to mess up release for
this. But we can get all our birds in a line to do it right after
> If we like the idea, I'll start looking into the fedora-messaging based
> consumer, using the approach from the current service, it should be pretty
> straight forward (I'll re-use the same project for it).
So this would just listen for commit / pr merge and do a git pull /
Listens to commits in pagure.io and pull when it sees some in the ansible repo.