On Thu, Feb 14, 2013 at 01:13:12PM +0100, Jakub Hrozek wrote:
On Wed, Feb 13, 2013 at 07:34:54PM -0500, Dmitri Pal wrote:
> I was thinking a bit about the ticket
> AFAIU the problem is that in the case when there are many netgroups they
> are expiring at the same time and every so often the system has to
> refetch all of them.
> The ticket calls for a job that would run asynchronously and refresh
> things that already expired. While this might be useful it seems like a
> big task. I was wondering about a different "complementary" approach
> that might help with making entries not expire at the same time. Would
> it be helpful it we as an option parameter something like:
> <map>_extended_expiration_range = %of_the_cache_lifetime.
> If not defined it will work as right now. If defined (per map) it will
> set the window of expiration. Here is how it would work.
> Say the expiration time is 10h and midpoint cache is at 50%. Without
> activity after 10 hours all entries would expire and the user would
> experience a delay. If the new parameter is defined say 100% the
> expiration timestamp will be set in between 10h and 10h + 10h * 100% =
> 20h randomly. It does not solve the problem of the refresh after 20h if
> there is no activity happening, but if there is some minimal activity
> happening within 10h and 20h window not all netgroups would expire at
> the same time and not all of them need to be refreshed at the same time.
> Does this make sense? Would it make things better?
I was experimenting with the problem this morning as I know this is a
big pain point for a customer.
I think the solution can be actually much simpler:
Right now, the midpoint refresh works if the expiration is N percent
into the cached entry timeout BUT not past the cache validity.
I think the simplest solution would be to either add a new option or
extend the current one to always return from cache even if the entry is
past its cache validity and schedule a refresh in the background. The
downside would obviously be that if the entry was removed from the LDAP
server then the first lookup would always hit it (unless it was scooped
by the cleanup task).
We discussed this problem at length on the phone with Sumit, Simo and
Dmitri. The conclusion was that my proposal was not acceptable b/c if
there were long delays between accessing the data, the user might
receive data created days or even weeks ago, which is not acceptable.
We'll use the periodic refresh task Simo proposed.