Hi Larry,
I have not reviewed any of your work yet, but I have some
comments about this effort. Many years ago we all spent a lot of
time trying to use e_poll through something we called nunc-stans.
It was a mess, and we were never able to resolve all the issues we
found (FD leaks, lost connections, instability, high CPU, loops,
etc). You can see this code in the 1.4.1 branch I believe (as it
was stripped out in later releases).
Now the connection code is very fragile, and any changes will
need extensive testing, that being said, the 1.4.4 branch is
essentially dead (especially in regards to major RFEs). Any work
you are doing should be done on master branch (which will be
2.1.0), because any major changes to the connection code will not
be backported to anything earlier (sorry).
Anyway, I'm not trying to be negative. This is exciting work,
and it's something that we had wanted to go back and revisit. I'm
sure William and Thierry will have some comments about some of the
challenges we faced, and some testing scenarios to try. But you
will need to port this to master branch, so I suggest getting off
of 1.4.4 asap.
Thanks again for working on this, and we will help you as much as
we can!
Cheers,
Mark
On 2/1/22 5:20 PM, Larry Lile wrote:
Hi,
I have been working on
converting slapd from using NSPR PR_Poll to using epoll(7),
forked from release 1.4.4.
I would appreciate any
feedback from the community on my progress so far and any
assistance with bringing this change to completion. I also
hope that it might well be integrated with James Chapman's
Connection Table splitting proposal and further proposal
regarding listener threading.
I believe my code still
contains an error, causing it to occasionally lose track of a
connection under heavy load, but I have so far been unable to
find the error. It doesn't seem to happen when I have
logging at SLAPI_LOG_CONNS, so it is possible I have caused or
encountered a race condition.
I tried not to deviate too
far from the existing code at this point, the major changes at
this point are:
- Listeners moved to a listen_table
(setup_epoll_listen_pds)
- listen_table is a list of Connection's so the can be
handled in the same way as a client Connection
- Differentiated by Connection->conn_state =
CONN_STATE_LISTEN
- Connection_Table->listen_count is no longer
maintained
- Eliminates listener pd handing from setup_pr_read_pds
- epoll_arm_listen_pds
- Adds or removes all listener pds from epoll
- Triggered from main event loop based on connection
count limits
- Connection_Table->fd is currently only maintained for
listeners
- This could likely be fully eliminated, but I'm not
sure what to do with signalpipe to accomplish this end
- Connection_Table->epollfd has been added to hold the
epoll fd set
- handle_new_connection
- Adds descriptors to epoll immediately
- Eliminates the need for setup in setup_pr_read_pds
- epoll_pr_idle_pds ( timeout related section of
setup_pr_read_pds )
- Should only handle client timeouts or special cases
for re-adding a descriptor to epoll
- Eliminates the needs from the remainder of
setup_pr_read_pds
- setup_pr_read_pds is not used with epoll
Is epoll(7) available
on all platforms supported by 389-ds? Because I don't
know, I have hesitated to remove any NSPR related code at
this point.
In my testing I have found
that epoll is provides a measurable boost in client
servicing, however my current testing methodology is not
sufficiently regimented enough to provide statistically
sound measurements.
I believe conversion from
PR_Poll to epoll(7) fits well with the "389 ds connection
management proposal" that James Chapman had raised.
When epoll is accepting a
large number of concurrent connections there are obvious
stalls that indicate the need for one or more listener
threads to be created to separate client connection
processing from connected client servicing.
I also think that James'
idea of creating multiple Connection Tables could be
simplified with epoll.
- Connection_Table->epollfd could be converted to an
array of epoll fd sets
- one thread and epoll fd for each listener
- one thread and epoll fd for each "Connection Table"
processors
- Re-balancing connections between "Connection Table"
processors could then be accomplished by adding and
deleting the fd in the appropriate "Connection Table"
epoll fd sets
Thanks in advance for all
input or assistance.
_______________________________________________
389-devel mailing list -- 389-devel@lists.fedoraproject.org
To unsubscribe send an email to 389-devel-leave@lists.fedoraproject.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/389-devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
--
Directory Server Development Team