Hello!
I was chatting with puiterwijk this morning (well, I guess his afternoon
☺) about the difficult week the systems team has had. There were quite a
few changes all happening at once this week:
* The update/reboot cycles
* RHEL 7.4 was released
* Pagure over dist-git was turned on
* Bodhi was reconfigured to talk to Pagure instead of Pkgdb
* A fedmsg update
* More things?
Patrick told me that it's been a huge amount of work for him and I
assume others this week, because many of the above didn't go so smoothly.
I think it's expected and frankly normal for updates to not go smoothly
- we are all stretched pretty thin and we don't have a formal QE team to
make sure that our infra apps are ready to ship. Thus, I think we should
expect changes like the above to be "bumpy" and instead should develop a
plan to help smooth out the bumps.
I propose that we use our infra calendar (or even a new calendar if
preferred) to more formally schedule infrastructure changes, with the
goal being to avoid weeks like this one (where so many large changes
landed at once). i.e., if I want to make a Bodhi deployment on Monday
and I see that there's a Pagure deployment already scheduled for Monday,
I should consider a different day.
Of course, we can always have exceptions. Sometimes we might have "flag
days" where we do want/need two apps to upgrade together. Or maybe
sometimes we do want to take advantage of an update/reboot cycle to
sneak in an app upgrade, if the systems team is comfortable of course.
That's fine when needed, but when not needed I think spreading out the
changes will help our systems team avoid insane weeks like this one.
Thoughts?