On 10/23/18 7:31 PM, Michael McLean wrote:
On Tue, Oct 23, 2018 at 7:35 PM Mátyás Selmeci
<matyas(a)cs.wisc.edu
<mailto:matyas@cs.wisc.edu>> wrote:
Yeah, I'll raise the postgres limit to 200. Thing is, it's
worked for years at 25, so I'm worried that it's a symptom of a
larger issue.
Hmm. I'm seeing frequent rapid accesses by kojira in
ssl_access_log. I wonder if that's it...
kojira is unlikely to be the culprit. It is only one daemon. It does
have a few threads that each connect, so it's consuming up to three
connections, but that's a far cry from 100. Sure, there are lots of
calls, but mostly serialized.
You might try setting LogLevel = INFO in hub.conf and looking at the
call records in ssl_error_log (and/or error_log). Those will have a
bit more info than the access logs.
Otoh, perhaps your usage has grown slowly over the years and just
now reached a tipping point?
Perhaps? I restarted Kojira to add a debugging line and the number of
connections dropped down to 7--10 and is still holding steady at that
range, even while I was doing a build and it was regenerating the repos.
I'll leave things going as-is for a while and see if the issue comes back.
-Mat
On 10/23/18 12:17 PM, Michael McLean wrote:
> Can you raise the limit in postgres? Optionally, lower
> MaxClients in httpd (that shifts the problem around a bit, but
> most Koji clients, e.g. builders, should retry if httpd refuses
> connection).
>
> Maybe crank up kojihub logging and try to see what client is
> overloading your hub with rapid calls?
>
> How many builders do you have? How many users?
>
> On Tue, Oct 23, 2018 at 12:40 PM Mátyás Selmeci
> <matyas(a)cs.wisc.edu <mailto:matyas@cs.wisc.edu>> wrote:
>
> Hi,
>
> Recently we've run into a problem where Koji uses far too
> many database
> connections and run over the simultaenous connections limit
> (of 100).
> The connections only last for a few seconds. This has
> caused lots of
> temporary "Database Outage" errors. I'd hate to keep
> bumping the
> connection limit since I feel like that's just papering
> over the
> problem.
>
> The specific error I see in the postgres log is:
>
> FATAL: connection limit exceeded for non-superusers
>
> I also see lots of the following:
>
> LOG: unexpected EOF on client connection
>
> Does anyone know what might be going on? This setup has
> been working
> fine for years and the problem has only shown up a few
> months ago.
>
> Unfortunately we're still on Koji 1.11.0.
>
> Thanks,
> -Mat
>
>
> --
> Mátyás (Mat) Selmeci
> Open Science Grid Software Team / Center for
> High-Throughput Computing
> University of Wisconsin-Madison Department of Computer Sciences
>