Greetings all,
I have recently installed koji(-hub/web) 1.27.0 on CentOS 8 stream and I'm seeing some very odd behavior. It appears that koji is maxing out the connections to the postgres database. I'm running postgres 13 with all default connection settings (max_connections=100) using unix-sockets instead of tcp-sockets.
Periodically, clicking around in the koji-web interface yields the message "Server is offline". After some time I am able to resume clicking links.
Postgres logs show this message when the "Server offline" message appears:
2022-01-07 15:41:48.449 CST [185432] FATAL: remaining connection slots are reserved for non-replication superuser connections 2022-01-07 15:43:09.907 CST [185558] FATAL: remaining connection slots are reserved for non-replication superuser connections
Systemd shows the following: ● postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-01-07 17:25:12 CST; 3min 8s ago Process: 199845 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS) Main PID: 199848 (postmaster) Tasks: 100 (limit: 75288) Memory: 201.8M CGroup: /system.slice/postgresql.service ├─199848 /usr/bin/postmaster -D /var/lib/pgsql/data ├─199849 postgres: logger ├─199851 postgres: checkpointer ├─199852 postgres: background writer ├─199853 postgres: walwriter ├─199854 postgres: autovacuum launcher ├─199855 postgres: stats collector ├─199856 postgres: logical replication launcher ├─199867 postgres: koji koji [local] idle ├─199869 postgres: koji koji [local] idle ├─199870 postgres: koji koji [local] idle ├─199871 postgres: koji koji [local] idle ├─199872 postgres: koji koji [local] idle ...
The host that runs koji has only been active for one day and is not configured with any tags, buildroots, or packages. There are only two build hosts connected to the hub.
I would appreciate any assistance you could give.
PS - I increased the max_connections setting in postgres to 200 and eventually that limit is also reached.
How does it map to apache's ServerLimit, etc. values? Is it possible that there is a much higher value of allowed httpd processes to be alive in the same moment? Having 300 max_connections is quite enough for large-scale deployment, so I would more look into how apache is configured than tuning the postgres.
On Sat, Jan 8, 2022 at 1:20 AM Robby Callicotte rcallicotte@mailbox.org wrote:
Greetings all,
I have recently installed koji(-hub/web) 1.27.0 on CentOS 8 stream and I'm seeing some very odd behavior. It appears that koji is maxing out the connections to the postgres database. I'm running postgres 13 with all default connection settings (max_connections=100) using unix-sockets instead of tcp-sockets.
Periodically, clicking around in the koji-web interface yields the message "Server is offline". After some time I am able to resume clicking links.
Postgres logs show this message when the "Server offline" message appears:
2022-01-07 15:41:48.449 CST [185432] FATAL: remaining connection slots are reserved for non-replication superuser connections 2022-01-07 15:43:09.907 CST [185558] FATAL: remaining connection slots are reserved for non-replication superuser connections
Systemd shows the following: ● postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-01-07 17:25:12 CST; 3min 8s ago Process: 199845 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS) Main PID: 199848 (postmaster) Tasks: 100 (limit: 75288) Memory: 201.8M CGroup: /system.slice/postgresql.service ├─199848 /usr/bin/postmaster -D /var/lib/pgsql/data ├─199849 postgres: logger ├─199851 postgres: checkpointer ├─199852 postgres: background writer ├─199853 postgres: walwriter ├─199854 postgres: autovacuum launcher ├─199855 postgres: stats collector ├─199856 postgres: logical replication launcher ├─199867 postgres: koji koji [local] idle ├─199869 postgres: koji koji [local] idle ├─199870 postgres: koji koji [local] idle ├─199871 postgres: koji koji [local] idle ├─199872 postgres: koji koji [local] idle ...
The host that runs koji has only been active for one day and is not configured with any tags, buildroots, or packages. There are only two build hosts connected to the hub.
I would appreciate any assistance you could give.
PS - I increased the max_connections setting in postgres to 200 and eventually that limit is also reached.
-- Robby Callicotte He/Him/His Timezone: America/Chicago IRC: c4t3l | Twitter: @robbycl2v_______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Monday, January 10, 2022 4:48:15 AM CST Tomas Kopecek wrote:
How does it map to apache's ServerLimit, etc. values? Is it possible that there is a much higher value of allowed httpd processes to be alive in the same moment? Having 300 max_connections is quite enough for large-scale deployment, so I would more look into how apache is configured than tuning the postgres.
Looks like my apache was mis-configured... It was set to use the mpm-event worker. I simply switched over to plain old prefork and all is well.
Thank you Tomas!
Yeah, in my small local test environments, I've had to set max_connections to 500, like so:
https://github.com/ktdreyer/koji-playbooks/blob/93068bb12e249703bc08c96e62b5...
I guess we need to update the server admin guide as a stop-gap measure, and longer-term, test out Mike's suggested psycopg2.pool branch.
- Ken
- Ken
On Fri, Jan 7, 2022 at 7:19 PM Robby Callicotte rcallicotte@mailbox.org wrote:
Greetings all,
I have recently installed koji(-hub/web) 1.27.0 on CentOS 8 stream and I'm seeing some very odd behavior. It appears that koji is maxing out the connections to the postgres database. I'm running postgres 13 with all default connection settings (max_connections=100) using unix-sockets instead of tcp-sockets.
Periodically, clicking around in the koji-web interface yields the message "Server is offline". After some time I am able to resume clicking links.
Postgres logs show this message when the "Server offline" message appears:
2022-01-07 15:41:48.449 CST [185432] FATAL: remaining connection slots are reserved for non-replication superuser connections 2022-01-07 15:43:09.907 CST [185558] FATAL: remaining connection slots are reserved for non-replication superuser connections
Systemd shows the following: ● postgresql.service - PostgreSQL database server Loaded: loaded (/usr/lib/systemd/system/postgresql.service; enabled; vendor preset: disabled) Active: active (running) since Fri 2022-01-07 17:25:12 CST; 3min 8s ago Process: 199845 ExecStartPre=/usr/libexec/postgresql-check-db-dir postgresql (code=exited, status=0/SUCCESS) Main PID: 199848 (postmaster) Tasks: 100 (limit: 75288) Memory: 201.8M CGroup: /system.slice/postgresql.service ├─199848 /usr/bin/postmaster -D /var/lib/pgsql/data ├─199849 postgres: logger ├─199851 postgres: checkpointer ├─199852 postgres: background writer ├─199853 postgres: walwriter ├─199854 postgres: autovacuum launcher ├─199855 postgres: stats collector ├─199856 postgres: logical replication launcher ├─199867 postgres: koji koji [local] idle ├─199869 postgres: koji koji [local] idle ├─199870 postgres: koji koji [local] idle ├─199871 postgres: koji koji [local] idle ├─199872 postgres: koji koji [local] idle ...
The host that runs koji has only been active for one day and is not configured with any tags, buildroots, or packages. There are only two build hosts connected to the hub.
I would appreciate any assistance you could give.
PS - I increased the max_connections setting in postgres to 200 and eventually that limit is also reached.
-- Robby Callicotte He/Him/His Timezone: America/Chicago IRC: c4t3l | Twitter: @robbycl2v_______________________________________________ koji-devel mailing list -- koji-devel@lists.fedorahosted.org To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.o... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
koji-devel@lists.fedorahosted.org