Mike McGrath wrote:
Toshio Kuratomi wrote:
> Mike McGrath wrote:
>> Bill Nottingham wrote:
>>> Toshio Kuratomi (a.badger(a)gmail.com) said:
>>>> Here's a short script to test our TG apps run via supervisor for
>>>> excessive memory usage and restart them if necessary. We could run
>>>> this via cron in alternate hours on each app server. Does this
>>>> seem like a good or bad idea to people?
>>>>
>>>
>>> It's a good idea if it's needed, but it's a bad idea that it is
>>> needed. What's
>>> wrong with TG that it leads to this situation?
>>>
>>
>> I was wondering this myself, I know smolt recently had some major
>> changes to keep memory usage down. Which TG apps are having this
>> issue and how often? I know MM uses a lot of memory but, AFAIK, it
>> was determined that there's not much of a leak if there is one and
>> that all of that memory is actually used.
>>
> Looks like smolt was upgraded just before Thanksgiving so it could be
> that we've plugged the leaks we had to deal with that inspired me to
> write this. Would it be a good idea to have this in place anyways?
> With it periodically checking, we would find out that we had problems
> when cron emails us a notice that the script had to restart a process.
> Without it, we'll be notified when nagios or a user tells us they're
> getting timeouts.
I think its a good idea if for no other reason then allows us to more
actively monitor this stuff, we'll get notified when the app restarts.
+1 from me with the intention that, over time, we get fewer and fewer
restarts.
Cool. I'll check it in and set up a cron job.
One further piece of information since I have output from testing this
on app3 yesterday:
AppName Uptime RSS 11/25 RSS 11/26
mirrormanager 2d4h 714336 962268
packagedb --- restarted 13h ago --
smolt 5d3h 299556 299556
transifex 5d3h 42744 42768
-Toshio