On Tue, Oct 23, 2018 at 7:35 PM Mátyás Selmeci <matyas(a)cs.wisc.edu> wrote:
Yeah, I'll raise the postgres limit to 200. Thing is, it's
worked for
years at 25, so I'm worried that it's a symptom of a larger issue.
Hmm. I'm seeing frequent rapid accesses by kojira in ssl_access_log. I
wonder if that's it...
kojira is unlikely to be the culprit. It is only one daemon. It does have
a few
threads that each connect, so it's consuming up to three connections,
but that's a far cry from 100. Sure, there are lots of calls, but mostly
serialized.
You might try setting LogLevel = INFO in hub.conf and looking at the call
records in ssl_error_log (and/or error_log). Those will have a bit more
info than the access logs.
Otoh, perhaps your usage has grown slowly over the years and just now
reached a tipping point?
Thanks,
-Mat
On 10/23/18 12:17 PM, Michael McLean wrote:
Can you raise the limit in postgres? Optionally, lower MaxClients in httpd
(that shifts the problem around a bit, but most Koji clients, e.g.
builders, should retry if httpd refuses connection).
Maybe crank up kojihub logging and try to see what client is overloading
your hub with rapid calls?
How many builders do you have? How many users?
On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci <matyas(a)cs.wisc.edu>
wrote:
> Hi,
>
> Recently we've run into a problem where Koji uses far too many database
> connections and run over the simultaenous connections limit (of 100).
> The connections only last for a few seconds. This has caused lots of
> temporary "Database Outage" errors. I'd hate to keep bumping the
> connection limit since I feel like that's just papering over the
> problem.
>
> The specific error I see in the postgres log is:
>
> FATAL: connection limit exceeded for non-superusers
>
> I also see lots of the following:
>
> LOG: unexpected EOF on client connection
>
> Does anyone know what might be going on? This setup has been working
> fine for years and the problem has only shown up a few months ago.
>
> Unfortunately we're still on Koji 1.11.0.
>
> Thanks,
> -Mat
>
>
> --
> Mátyás (Mat) Selmeci
> Open Science Grid Software Team / Center for High-Throughput Computing
> University of Wisconsin-Madison Department of Computer Sciences
> _______________________________________________
> koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
> To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
> Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
> List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
>
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
>
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://getfedora.org/code-of-conduct.html
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...