On 11/10/20 12:31 PM, Frantisek Zatloukal wrote:
Nevermind, I wouldn't have been able to sleep before diagnosing
it :) .
However, since we're running on a pretty exhausted and overloaded
temporary server, OOM sometimes kills some of our workers or even the
redis/database. In these cases, when we lose our tasks queue, but
counters stay incremented, we would end up not syncing these items
ever (with incremented counters but not in the queue).
To workaround this and further increase reliability, we have
implemented a watchdog. It simply goes throughout the cache for any
items which weren't synced when they should've been and resets the
counter for them. It runs once a day.
Thanks for the detailed response and quick fix! If it helps any: you can
use keyspace notifications to track eviction from memory. You might be
able to work with that and maxmemory to avoid oom more.