Change in vdsm[master]: AdvancedStatsThread: Throttle duplicated exception log

zhshzhou at linux.vnet.ibm.com zhshzhou at linux.vnet.ibm.com
Tue Oct 30 01:51:00 UTC 2012


Zhou Zheng Sheng has posted comments on this change.

Change subject: AdvancedStatsThread: Throttle duplicated exception log
......................................................................


Patch Set 1: (2 inline comments)

Thanks Yaniv and Mark. From the discussion above, I think messages from stats functions need a dedicate channel and filter.

....................................................
File vdsm/utils.py
Line 308: 
Line 309:     def needSuppress(self, statsFunction, exceptObj):
Line 310:         if (statsFunction not in self._exceptions or
Line 311:             not self._exceptionEqual(
Line 312:                     self._exceptions[statsFunction]['exception'], exceptObj)):
Thanks for noticing this problem. I thought of it too. The patch here is just for throttling exception log, so when a stats function raises a exception, it stops executing at that point, it means the problem remains the same, then it will raise the same exception next time, so I think the situation you mentioned could be very rare.

I think implementing a dedicate log channel to filter these messages is better.
Line 313: 
Line 314:             self._exceptions[statsFunction] = {'exception': exceptObj,
Line 315:                                                'dupCount': 0}
Line 316:             return False


Line 436:                         statsFunction()
Line 437:                     except Exception, e:
Line 438:                         if not self.handleStatsException(e):
Line 439:                             if not self._logThrottle.needSuppress(
Line 440:                                         statsFunction, e):
Even for this kind of serious messages, it's meaningless to report the same content again and again, and we do want to be alerted when the bug is triggered. So I think these messages can go to a dedicate channel, and the receiver side of the channel discard the duplicated messages and just keep a limited number of latest messages for the same kind. Or the messages can be sent to some supervisor program.

When we know exactly what the problem is, we can just grep the log and find out what we need. Sometimes we do not know problem, so just want to watch the log line by line. If this kind of messages are too much, the log is hard to digest. I think the log of stats functions both debug and error messages, can be put to a separated log file.  It's really annoying to see the debugging messaging of "dd" every second when 'tail -f' the vdsm log. Yes we can grep out the unwanted "dd" messages, but it's still annoying to add an extra pipe to grep out the "dd" messages every time.
Line 441:                                 self._log.error("Stats function failed: %s",
Line 442:                                                 statsFunction, exc_info=True)
Line 443: 
Line 444:             self._stopEvent.wait(waitInterval)


--
To view, visit http://gerrit.ovirt.org/8884
To unsubscribe, visit http://gerrit.ovirt.org/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I2ee04d8d82e2a14b0a003627981c28e5e64a46ab
Gerrit-PatchSet: 1
Gerrit-Project: vdsm
Gerrit-Branch: master
Gerrit-Owner: Zhou Zheng Sheng <zhshzhou at linux.vnet.ibm.com>
Gerrit-Reviewer: Federico Simoncelli <fsimonce at redhat.com>
Gerrit-Reviewer: Gal Hammer <ghammer at redhat.com>
Gerrit-Reviewer: Mark Wu <wudxw at linux.vnet.ibm.com>
Gerrit-Reviewer: Yaniv Bronhaim <ybronhei at redhat.com>
Gerrit-Reviewer: Zhou Zheng Sheng <zhshzhou at linux.vnet.ibm.com>


More information about the vdsm-patches mailing list