Below are the notes about Autocloud update happened last week. There
were few things which went bad, few points worked well. I am trying to
list them here, and how do we plan to make sure that the bad things
never happen again. I am ccing patrick in this mail as he was the
contact point from fedora infrastructure (read: he did all the work).
## deployment using ansible to the newly installed systems
We only had to change yum to dnf, and also had to remove the old hotfix
part from the roles. The current playbooks, roles seems to be stable,
and can help us in the future too.
## fedmsg-hub on the backends double enqueue
We had to restart fedmsg-hubs few times in the backends servers. In
between we also found that fedmsg-hub service was happily enqueueing the
jobs twice (for each compose), and then it got fixed automagically,
nothing was changed in our configuration or in code.
We are still not sure why this happened, but we are trying to dig more
## fedmsg-hub broke due to a faulty dependency
Even though we kept our code up and running for weeks, after the
production deployment we found one of the dependency (fedfind, adamw is
the upstream author) was broken with the fedora atomic image names, and
causing our fedmsg-hub instances go crazy. We have informed upstream,
and got a quick hotfix deployment in few hours after finding the issue.
For the next release we will make sure if keep it running for longer on
our internal hardware with messages from production fedmsg. This
dependency failure was something we should have caught, but could not.
## missing fedmsg(s) from autocloud on completing the testing of a compose
Now this was a known part of the whole development+release cycle. Sayan had
submitted the patch , but there was some slight miscommunication. In my part
i missed to track the state of this dependency.
For the next release, i will make a release/deployment checklist for
autocloud, and get it validated from everyone involved. Most probably we
will add too many minor details to this checklist, but that will help us
to keep things in track about any future deployment. sayan is currently
working to get that particular change in production so that we can send
out fedmsg(s) as required by adam.
## missing package dependency causing missing bridge on libvirt backend
We also found that a missing link in the dependency chain caused a
missing virtual bridge in the libvirt backend. Patrick helped to find
that adding libvirt as dependency for that particular box will fix the
issue in future.
We should test on clean installations while developing next time to make
sure this is not repeated. Plus we should think about getting better on
the stage environment.
## the new webfrontend is better
Sayan did a good job in making the new webfrontend. We can now point out
to the exact failures .
In future we should try to get more input about the features of the
webfrontend. Even though the whole service is made for automation, but
this frontend helps us to find, and point to the right issues found in
## autocloud+tunir did what they are supposed to do
After fixing the hiccups, the autocloud service is doing what it
supposed to do, testing the images. We will push our effort in having
better test coverage in the coming months to take the advantage from
this new deployment.
Please comment/suggest whatever you think about the work. This will help
us to improve in the future releases.
Fedora Cloud Engineer
CPython Core Developer