On 10/06/2011 03:11 PM, Ayal Baron wrote:
>> I would like to propose enhancing the VDSM source to emit >> message >> IDs in >> log messages with levels greater than DEBUG (i.e. INFO, ERROR, >> WARN, >> EXCEPTION, and CRITICAL). Messages IDs have been used >> successfully in >> many products and they can aid in debugging, documentation, >> root >> cause >> analysis, and internationalization. >> >> I have scanned the source and have determined that there are >> roughly 525 >> calls to log.[error|warn|info|critical|exception] and I am >> volunteering >> to go through and add IDs to them all but, before I do I want >> to get >> buy-in from the project maintainers. While adding IDs to the
wrt i18n, I'm not entirely convinced.
Yeah, i18n is one reason to do it but it is only relevant if you take the time to create lang-packs for different nationalities (which can be expensive). However, it is a cakewalk to code for in Python _("string here"). So you can code for it ... and if someone ever wanted to take the time create a lang-pack you'd be set.
Every line we log has the module name and line (never localized) which logged it so effectively we already have this info. However, the ID would remain constant where lines will change place between versions so it makes it easier. I also agree about the lookup and being able to write simpler parsers.
So I'm in.
Cool.
>> messages I >> would also enable them for i18n. >> >> I have written a rough plan for adding messages IDs to VDSM >> below >> and >> I've provided some examples of products/projects that use >> message >> IDs. >> >> Examples of Projects that Use Message IDs: >> - JBoss 7 is adopting Message IDs for all of its logging. See >> the >> annotations (e.g. @Message(id = 12000, ) in the following >> piece of >> source: >> https://github.com/jbossas/jboss-as/blob/master/process-controller/src/main/... >> >> >> >> - Almost all IBM products use message IDs and they do a fairly >> good >> job >> of documenting them. Google has indexed them all and you can >> simply >> type >> them in to a search and get a description of the problem that >> is >> associated with the ID. >> http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.omegam... >> >> >> >> >> >> Example of a Log Statement with a Message ID in Python: >> FORMAT = '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >> %(message)s' >> logging.basicConfig(format=FORMAT) >> logger = logging.getLogger() >> logger.warning('NFS problem. Unable to mount.', extra={ >> 'messageid' : >> 'VDSM1000'}) >> >> // Produces >> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS problem. >> Unable >> to mount. >> >> >> Example of a Log Statement with a Message ID in Python with >> i18n: >> logger.warning(_('NFS problem. Unable to mount.'), extra={ >> 'messageid' : >> 'VDSM1000'}) >> >> Example of an i18n Log Message: >> Chinese: 2011年10月5日13:24:43,18011警告(VDSM1000):NFS问题。无法 >> 安 装。 >> English: 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS >> problem. >> Unable to mount. >> >> >> >> A Rough Plan for adding Message IDs to VDSM: >> >> Step 0: Decide on a logging format of the ID itself: >> - JBoss is using "JBAS#####". I am thinking that VDSM could >> probably do >> something similar "VDSM####". >> >> Step 1: Decide where to list and document all of the IDs. >> - It could be as simple as a page on VDSM's fedorahosted wiki. >> >> Step 2: Pin the format of the log messages: >> - Currently VDSM allows a user configurable Python log format. >> We >> would >> need to pin the format so that it includes a substitution >> variable for >> the message ID. >> Example: '%(asctime)s %(lineno)s %(levelname)s(%(messageid)s): >> %(message)s' >> >> Step 4: Plow through the VDSM source and add message IDs. >> >> >> Cheers, >> Keith >> >> >> >> >> >> _______________________________________________ >> vdsm-devel mailing list >> vdsm-devel@lists.fedorahosted.org >> https://fedorahosted.org/mailman/listinfo/vdsm-devel > Seems like a nice idea. But how does it affect developers. How > do > they choose mIDs? How do you prevent collisions? How are they > different from simple gettext?
As I never used this myself so I can't really imagine the all benefits of using such a system, at least enough to warrant all the tedious work. If you could just list out some of the use cases it'll be most helpful.
For instance I assume grepping for an ID is easier then grepping for a message especially in multiple versions where the message might have changed.
Sure. As far as the tedious part, yes, adding message IDs to an existing source tree is a bit of a PITA but the 525 invocations in VDSM isn't all that bad. I've seen much worse. Plus, once it's done the effort is much smaller (new lines of code only).
As for the value (I've tried to itemize a few common reasons for doing it):
Value: Message IDs allow you to use gettext globalize your messages. If you just used gettext and pulled the strings out of your code for globalization then you might end up with something like this in vdsm.log... 错误:问题安装驱动器 the English translation (according to google) is: ERROR: Problem mounting drive Unless you can read Chinese, then the message above is completely unhelpful to a support engineer. The converse is true also. Consider a non-english speaking person trying to decipher "ERROR: Problem mounting drive". With a message ID; however, you get this... 错误(VDSM1000):安装驱动问题
Is it possible to just hash the main string before internationalization?
So basically:
fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str, crc32(message), message);
Yes, a hash of the string is possible. However, would make it hard for someone to document the ID because they would have to hash the string to get the ID.
One of the features of message ID/codes is that they are documentable.
Or the equivalent Python.
Value: Message IDs allow other applications to perform actions based on what is in your logs. On the mainframe, this is extremely common. You'll have one program watching the logs for a particular ID(s). When the program sees the ID it will look in a dictionary and decide what (if anything) needs to be done. On linux, logwatch, is frequently used for this purpose and message IDs make this process much simpler because the program doesn't have to write complicated REGEXs to look for arbitrary strings it can just scan for an ID.
Value: Message IDs generally have the same meaning across releases of the product. This means that the developer can change the text of the log message without worrying about breaking someone's automation. Example: Version 1 of the product- ERROR(VDSM1000): Problem mounting drive. Version 2 of the product- ERROR(VDSM1000): unable to contact 127.0.0.1 for mounting. Try blah blah.
Does this mean that logging messages are an ABI that needs to be maintained over time?
The degree to which message IDs need to be maintained is generally a function of how dynamic the code is and how much you want to document what the ID actually means. If, on the WIKI wherein you define messages, you elect to say [1] then this "ABI" doesn't need any maintenance whatsoever.
[1] VDSM1000 == "Call support or post to the forum to figure out what the heck the program just did"
That's probably very difficult to do right.
Lots of other software programs have been successful at it [1], [2]. I would prefer to think of it in terms of how easy we want to make it for people to self diagnose. Also, without message IDs supporting source that is i18n is nearly impossible unless everyone is multilingual.
[1] http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_refe... [2] http://mail.google.com/support/bin/answer.py?answer=12336
Regards,
Anthony Liguori
Value: If you do it right search engines will index your message IDs and take users to a page where they can hopefully get more detail on the issue and potentially "self solve" the problem without calling support. Put this into a search engine... "KDFAP0020S"
Value: Message IDs allow you to easily describe flows. You might be able to say... "if you saw ID123 then look farther back for ID124 and if you see ID124 then the problem is _probably_ XYZ."
I am probably not doing enough justice to the value of message IDs but, I will say that lots of very large companies use them in complex software applications to solve these and other issues.
Cheers, Keith
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel