On Fri, Oct 11, 2019 at 09:22:11AM -0400, Stephen John Smoogen wrote:
On Fri, 11 Oct 2019 at 09:19, Peter Robinson
> On Fri, Oct 11, 2019 at 10:13 AM Dominik 'Rathann' Mierzejewski
> <dominik(a)greysector.net> wrote:
> > On Friday, 11 October 2019 at 04:36, Orion Poplawski wrote:
> > > On 10/10/19 4:57 PM, Orion Poplawski wrote:
> > > > On 10/10/19 9:23 AM, Kevin Fenzi wrote:
> > [...]
> > > > > Pretty please can you all indicate approximately when (UTC) you
> > > > > seeing the slowness? I suspect it may be the nightly database
> > > > > but I can't tell without knowing when you saw the
> > [...]
> > > > Thanks, it is much better now. I feel like I've hit periods of
> > > > over several days recently so I will definitely note times if I run
> > > > it again.
> > >
> > > Definitely slower again:
> > > Fri Oct 11 02:20:53 UTC 2019
> > Fri 11 Oct 09:11:01 UTC 2019
> > It takes 30s to show a build status page, e.g.
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=38215205
> Builds/cli is slow on Fri 11th, 13:17 UTC.
The fixes we tried seems to have put the load average of koji up to
150 while several mbs builds are going. I am going to put our status
as yellow and see if we can tune this down.
So to update this thread some:
* Load is fine/responsiveness is fine as long as we aren't doing a
* When we do, things slow down and load goes way up.
We did find one very odd/bad thing on the virthost the server is on (one
drive in the raid is showing active in multipath and possibly getting
copies of all i/o sent to it? unclear exactly, but it's wrong and needs
So, we stopped the backup and we are going to wait until 21UTC
(after 5pm EDT). At that point we will try and clean up that multipath
issue and see if that helps things. If not we will try and tune the
vm/database more until we have it responsive all the time.
We hope we can do these changes with minimal to no downtime, but there
may be some.
I've filed https://pagure.io/fedora-infrastructure/issue/8292
issue, folks are welcome to follow along there.
Sorry about all this, I was just hoping to make things faster and sadly,