Hello everyone,
People have noticed that they aren't getting some FMN notifications, so I spent the afternoon tracking down the issue. It looks as though the redis instance on notifs-backend01.phx2.fedoraproject.org is configured to persist to disk (this is apparently the default config, which is surprising to me). In addition, dogpile.cache is apparently not setting a TTL on keys despite being configured to do so.
All this means that everything in the FMN redis cache is _really_ stale and contains invalid objects. For example, the packages a user has ACLs on is cached as a set currently, but the latest version of FMN expects a dictionary. This leads to messages not getting sent that should have been sent.
I would like to delete the persisted database (/var/lib/redis/dump.rdb) and restart redis to wipe the cache. FMN does not like redis going down so the workers and backend will likely need to be restarted, and because of a different caching issue[0] this will likely take around an hour (messages won't be lost, they'll just build up in the rabbitmq queue).
[0] https://github.com/fedora-infra/fmn/pull/190
+1s?
On 05/17/2017 03:27 PM, Jeremy Cline wrote:
Hello everyone,
People have noticed that they aren't getting some FMN notifications, so I spent the afternoon tracking down the issue. It looks as though the redis instance on notifs-backend01.phx2.fedoraproject.org is configured to persist to disk (this is apparently the default config, which is surprising to me). In addition, dogpile.cache is apparently not setting a TTL on keys despite being configured to do so.
All this means that everything in the FMN redis cache is _really_ stale and contains invalid objects. For example, the packages a user has ACLs on is cached as a set currently, but the latest version of FMN expects a dictionary. This leads to messages not getting sent that should have been sent.
I would like to delete the persisted database (/var/lib/redis/dump.rdb) and restart redis to wipe the cache. FMN does not like redis going down so the workers and backend will likely need to be restarted, and because of a different caching issue[0] this will likely take around an hour (messages won't be lost, they'll just build up in the rabbitmq queue).
[0] https://github.com/fedora-infra/fmn/pull/190
+1s?
+1 here. Do it.
kevin
On Wed, May 17, 2017 at 03:38:25PM -0600, Kevin Fenzi wrote:
On 05/17/2017 03:27 PM, Jeremy Cline wrote:
Hello everyone,
People have noticed that they aren't getting some FMN notifications, so I spent the afternoon tracking down the issue. It looks as though the redis instance on notifs-backend01.phx2.fedoraproject.org is configured to persist to disk (this is apparently the default config, which is surprising to me). In addition, dogpile.cache is apparently not setting a TTL on keys despite being configured to do so.
All this means that everything in the FMN redis cache is _really_ stale and contains invalid objects. For example, the packages a user has ACLs on is cached as a set currently, but the latest version of FMN expects a dictionary. This leads to messages not getting sent that should have been sent.
I would like to delete the persisted database (/var/lib/redis/dump.rdb) and restart redis to wipe the cache. FMN does not like redis going down so the workers and backend will likely need to be restarted, and because of a different caching issue[0] this will likely take around an hour (messages won't be lost, they'll just build up in the rabbitmq queue).
[0] https://github.com/fedora-infra/fmn/pull/190
+1s?
+1 here. Do it.
+1 for me to
Pierre
infrastructure@lists.fedoraproject.org
-
Jeremy Cline -
Kevin Fenzi -
Pierre-Yves Chibon