On Fri, Feb 14, 2020 at 03:54:48PM +0100, Aurelien Bompard wrote:
I thought I'd make a summary of where I'm at. Here are the issues I found
and what I did about it:
- We ran into an Ansible issue that the PR
fixes. I've asked pingou to
patch batcave since it's basically a one-liner that will keep working with
the older prod version.
- When starting a RabbitMQ cluster from scratch, there is a race condition
that is documented here:
On nodes 02 and 03, I've just destroyed the database and let it
auto-detect the cluster again
# systemctl stop rabbitmq-server && rm -rf /var/lib/rabbitmq/mnesia/
systemctl start rabbitmq-server
It worked fine. I checked with "rabbitmqctl list_users" that all nodes
had the same users declared.
- I've also fixed a couple things in the playbooks that assumed the cluster
to be up and setup already.
- I've rebuilt collectd-rabbitmq for EPEL8 but we currently only install it
on production apparently (not sure why, I think it could be useful in
I think that was me disabling it in stg because it wasn't working?
- The nagios-plugins-rabbitmq RPM still fails to install because of
dependency bug in perl-Monitoring-Plugin, I've opened a ticket about it:
Now, we need to recreate the queues, users and bindings, and I don't have
the permissions to run all the playbooks. If someone could run the master
playbook limited on staging and on the rabbitmq_cluster tag, I think it
should recreate all users and queues and we should be all set.
I'm around and on IRC if you need me.
Thanks much for working on this. :)
I guess the next step is to add the stuff that we needed this new
version for and confirm it works? Then on to production?