mprahl added a new comment to an issue you are following:
``
The mass rebuild for modules failed because of architectural issues in MBS that become
apparent under high load, such as a mass rebuild. One such issue is that MBS only supports
one worker that processes messages, so when MBS is under high load, it takes several
minutes to an hour for it to start processing new messages. MBS has a separate process
called the "poller" which will try to detect missed messages. Under these
circumstances, it will assume a message has been lost and trigger it again even though MBS
may have just not gotten to it yet. This causes very weird conditions that make debugging
failed builds difficult. These issues are being addressed by issue 1311 [1], which will be
worked on soon by the team and will be done before December 1st.
The other issue is that MBS has an internal Python queue which stores all the messages it
receives. If MBS is restarted, all those messages are lost. The fedmsg-hub service crashed
due to the file descriptor leak, so all those messages were lost and the poller wasn't
smart enough to recover from there. The Python queue issue is being addressed by issue
1311 [1], but the fedmsg-hub issued is not currently planned and is tracked in issue
1234.
My suggestion is to cancel all existing builds that are stuck, and start the mass rebuild
again on ones that didn't succeed, but do it at a much slower pace. This can be done
by checking the number of module builds that are active, or sleeping an arbitrarily amount
between builds.
I'm sorry about the inconvenience, but the team is working on addressing the issue.
1 -
https://pagure.io/fm-orchestrator/issue/1311
2 -
https://pagure.io/fm-orchestrator/issue/1234
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/8664