OK, unless anyone has anything else to add I'm going to assume
that
the
project maintainers are OK with message IDs and I'll start the
process.
To summarize:
1. I'm going to do some minor surgery to the logger so that the log
format is pinned an not user modifiable. This is necessary to ensure
that message IDs can be substituted into the string.
2. Message IDs will have the following format: VDSM#####
3. Message IDs will just be a simple up-counter across all of VDSM.
4. Existing strings will be converted to translatable strings. The
string itself won't be changed it will just be wrapped by _(...) so
that
gettext will work.
5. Message IDs may be documented, somewhere not sure where yet, with
a
brief explanation (if an explanation is appropriate). This part
might
take some and; hopefully, the explanations will evolve. I expect
many
of the IDs to not have explanations right away though and I
definitely
don't want to put any bad explanations up or ones that haven't been
vetted.
Cheers,
Keith
On 10/06/2011 03:29 PM, Keith Robertson wrote:
> On 10/06/2011 03:11 PM, Ayal Baron wrote:
>>>>>>>>> I would like to propose enhancing the VDSM source to
emit
>>>>>>>>> message
>>>>>>>>> IDs in
>>>>>>>>> log messages with levels greater than DEBUG (i.e.
INFO,
>>>>>>>>> ERROR,
>>>>>>>>> WARN,
>>>>>>>>> EXCEPTION, and CRITICAL). Messages IDs have been
used
>>>>>>>>> successfully in
>>>>>>>>> many products and they can aid in debugging,
documentation,
>>>>>>>>> root
>>>>>>>>> cause
>>>>>>>>> analysis, and internationalization.
>>>>>>>>>
>>>>>>>>> I have scanned the source and have determined that
there
>>>>>>>>> are
>>>>>>>>> roughly 525
>>>>>>>>> calls to log.[error|warn|info|critical|exception]
and I am
>>>>>>>>> volunteering
>>>>>>>>> to go through and add IDs to them all but, before I
do I
>>>>>>>>> want
>>>>>>>>> to get
>>>>>>>>> buy-in from the project maintainers. While adding
IDs to
>>>>>>>>> the
>> wrt i18n, I'm not entirely convinced.
> Yeah, i18n is one reason to do it but it is only relevant if you
> take
> the time to create lang-packs for different nationalities (which
> can
> be expensive). However, it is a cakewalk to code for in Python
> _("string here"). So you can code for it ... and if someone ever
> wanted to take the time create a lang-pack you'd be set.
>> Every line we log has the module name and line (never localized)
>> which logged it so effectively we already have this info.
>> However, the ID would remain constant where lines will change
>> place
>> between versions so it makes it easier.
>> I also agree about the lookup and being able to write simpler
>> parsers.
>>
>> So I'm in.
>>
> Cool.
>>>>>>>>> messages I
>>>>>>>>> would also enable them for i18n.
>>>>>>>>>
>>>>>>>>> I have written a rough plan for adding messages IDs
to VDSM
>>>>>>>>> below
>>>>>>>>> and
>>>>>>>>> I've provided some examples of products/projects
that use
>>>>>>>>> message
>>>>>>>>> IDs.
>>>>>>>>>
>>>>>>>>> Examples of Projects that Use Message IDs:
>>>>>>>>> - JBoss 7 is adopting Message IDs for all of its
logging.
>>>>>>>>> See
>>>>>>>>> the
>>>>>>>>> annotations (e.g. @Message(id = 12000, ) in the
following
>>>>>>>>> piece of
>>>>>>>>> source:
>>>>>>>>>
https://github.com/jbossas/jboss-as/blob/master/process-controller/src/ma...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> - Almost all IBM products use message IDs and they
do a
>>>>>>>>> fairly
>>>>>>>>> good
>>>>>>>>> job
>>>>>>>>> of documenting them. Google has indexed them all and
you
>>>>>>>>> can
>>>>>>>>> simply
>>>>>>>>> type
>>>>>>>>> them in to a search and get a description of the
problem
>>>>>>>>> that
>>>>>>>>> is
>>>>>>>>> associated with the ID.
>>>>>>>>>
http://publib.boulder.ibm.com/infocenter/tivihelp/v42r1/topic/com.ibm.ome...
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Example of a Log Statement with a Message ID in
Python:
>>>>>>>>> FORMAT = '%(asctime)s %(lineno)s
>>>>>>>>> %(levelname)s(%(messageid)s):
>>>>>>>>> %(message)s'
>>>>>>>>> logging.basicConfig(format=FORMAT)
>>>>>>>>> logger = logging.getLogger()
>>>>>>>>> logger.warning('NFS problem. Unable to
mount.', extra={
>>>>>>>>> 'messageid' :
>>>>>>>>> 'VDSM1000'})
>>>>>>>>>
>>>>>>>>> // Produces
>>>>>>>>> 2011-10-05 13:24:43,180 11 WARNING(VDSM1000): NFS
problem.
>>>>>>>>> Unable
>>>>>>>>> to mount.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Example of a Log Statement with a Message ID in
Python with
>>>>>>>>> i18n:
>>>>>>>>> logger.warning(_('NFS problem. Unable to
mount.'), extra={
>>>>>>>>> 'messageid' :
>>>>>>>>> 'VDSM1000'})
>>>>>>>>>
>>>>>>>>> Example of an i18n Log Message:
>>>>>>>>> Chinese: 2011年10月5日13:24:43,18011警告(VDSM1000):NFS问
>>>>>>>>> 题。无法
>>>>>>>>> 安 装。
>>>>>>>>> English: 2011-10-05 13:24:43,180 11
WARNING(VDSM1000): NFS
>>>>>>>>> problem.
>>>>>>>>> Unable to mount.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> A Rough Plan for adding Message IDs to VDSM:
>>>>>>>>>
>>>>>>>>> Step 0: Decide on a logging format of the ID
itself:
>>>>>>>>> - JBoss is using "JBAS#####". I am
thinking that VDSM could
>>>>>>>>> probably do
>>>>>>>>> something similar "VDSM####".
>>>>>>>>>
>>>>>>>>> Step 1: Decide where to list and document all of the
IDs.
>>>>>>>>> - It could be as simple as a page on VDSM's
fedorahosted
>>>>>>>>> wiki.
>>>>>>>>>
>>>>>>>>> Step 2: Pin the format of the log messages:
>>>>>>>>> - Currently VDSM allows a user configurable Python
log
>>>>>>>>> format.
>>>>>>>>> We
>>>>>>>>> would
>>>>>>>>> need to pin the format so that it includes a
substitution
>>>>>>>>> variable for
>>>>>>>>> the message ID.
>>>>>>>>> Example: '%(asctime)s %(lineno)s
>>>>>>>>> %(levelname)s(%(messageid)s):
>>>>>>>>> %(message)s'
>>>>>>>>>
>>>>>>>>> Step 4: Plow through the VDSM source and add message
IDs.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>> Keith
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> vdsm-devel mailing list
>>>>>>>>> vdsm-devel(a)lists.fedorahosted.org
>>>>>>>>>
https://fedorahosted.org/mailman/listinfo/vdsm-devel
>>>>>>>> Seems like a nice idea. But how does it affect
developers.
>>>>>>>> How
>>>>>>>> do
>>>>>>>> they choose mIDs? How do you prevent collisions? How
are
>>>>>>>> they
>>>>>>>> different from simple gettext?
>>>>>> As I never used this myself so I can't really imagine the
all
>>>>>> benefits
>>>>>> of using such a system, at least enough to warrant all the
>>>>>> tedious
>>>>>> work. If you could just list out some of the use cases
it'll
>>>>>> be
>>>>>> most
>>>>>> helpful.
>>>>>>
>>>>>> For instance I assume grepping for an ID is easier then
>>>>>> grepping
>>>>>> for a
>>>>>> message especially in multiple versions where the message
>>>>>> might
>>>>>> have
>>>>>> changed.
>>>>> Sure. As far as the tedious part, yes, adding message IDs to
>>>>> an
>>>>> existing source tree is a bit of a PITA but the 525 invocations
>>>>> in
>>>>> VDSM
>>>>> isn't all that bad. I've seen much worse. Plus, once
it's
>>>>> done
>>>>> the
>>>>> effort is much smaller (new lines of code only).
>>>>>
>>>>> As for the value (I've tried to itemize a few common reasons
>>>>> for
>>>>> doing it):
>>>>>
>>>>> Value: Message IDs allow you to use gettext globalize your
>>>>> messages. If
>>>>> you just used gettext and pulled the strings out of your code
>>>>> for
>>>>> globalization then you might end up with something like this in
>>>>> vdsm.log...
>>>>> 错误:问题安装驱动器
>>>>> the English translation (according to google) is:
>>>>> ERROR: Problem mounting drive
>>>>> Unless you can read Chinese, then the message above is
>>>>> completely
>>>>> unhelpful to a support engineer. The converse is true also.
>>>>> Consider a
>>>>> non-english speaking person trying to decipher "ERROR: Problem
>>>>> mounting
>>>>> drive". With a message ID; however, you get this...
>>>>> 错误(VDSM1000):安装驱动问题
>>>> Is it possible to just hash the main string before
>>>> internationalization?
>>>>
>>>> So basically:
>>>>
>>>> fprintf(f, "[%s] ERROR(VDSM%04x) %s", date_str,
crc32(message),
>>>> message);
>>> Yes, a hash of the string is possible. However, would make it
>>> hard
>>> for
>>> someone to document the ID because they would have to hash the
>>> string
>>> to
>>> get the ID.
>>>
>>> One of the features of message ID/codes is that they are
>>> documentable.
>>>> Or the equivalent Python.
>>>>
>>>>> Value: Message IDs allow other applications to perform actions
>>>>> based on
>>>>> what is in your logs. On the mainframe, this is extremely
>>>>> common.
>>>>> You'll have one program watching the logs for a particular
>>>>> ID(s).
>>>>> When
>>>>> the program sees the ID it will look in a dictionary and decide
>>>>> what (if
>>>>> anything) needs to be done. On linux, logwatch, is frequently
>>>>> used for
>>>>> this purpose and message IDs make this process much simpler
>>>>> because the
>>>>> program doesn't have to write complicated REGEXs to look for
>>>>> arbitrary
>>>>> strings it can just scan for an ID.
>>>>>
>>>>> Value: Message IDs generally have the same meaning across
>>>>> releases
>>>>> of
>>>>> the product. This means that the developer can change the text
>>>>> of
>>>>> the
>>>>> log message without worrying about breaking someone's
>>>>> automation.
>>>>> Example:
>>>>> Version 1 of the product- ERROR(VDSM1000): Problem
>>>>> mounting
>>>>> drive.
>>>>> Version 2 of the product- ERROR(VDSM1000): unable to
>>>>> contact
>>>>> 127.0.0.1 for mounting. Try blah blah.
>>>> Does this mean that logging messages are an ABI that needs to be
>>>> maintained over time?
>>> The degree to which message IDs need to be maintained is
>>> generally a
>>> function of how dynamic the code is and how much you want to
>>> document
>>> what the ID actually means. If, on the WIKI wherein you define
>>> messages, you elect to say [1] then this "ABI" doesn't need
any
>>> maintenance whatsoever.
>>>
>>> [1] VDSM1000 == "Call support or post to the forum to figure out
>>> what
>>> the heck the program just did"
>>>
>>>> That's probably very difficult to do right.
>>> Lots of other software programs have been successful at it [1],
>>> [2].
>>> I
>>> would prefer to think of it in terms of how easy we want to make
>>> it
>>> for
>>> people to self diagnose. Also, without message IDs supporting
>>> source
>>> that is i18n is nearly impossible unless everyone is
>>> multilingual.
>>>
>>> [1]
>>>
http://www.cisco.com/en/US/products/sw/netmgtsw/ps1982/products_command_r...
>>>
>>> [2]
http://mail.google.com/support/bin/answer.py?answer=12336
>>>
>>>
>>>> Regards,
>>>>
>>>> Anthony Liguori
>>>>
>>>>> Value: If you do it right search engines will index your
>>>>> message
>>>>> IDs and
>>>>> take users to a page where they can hopefully get more detail
>>>>> on
>>>>> the
>>>>> issue and potentially "self solve" the problem without
calling
>>>>> support.
>>>>> Put this into a search engine... "KDFAP0020S"
>>>>>
>>>>> Value: Message IDs allow you to easily describe flows. You
>>>>> might
>>>>> be
>>>>> able to say... "if you saw ID123 then look farther back for
>>>>> ID124
>>>>> and if
>>>>> you see ID124 then the problem is _probably_ XYZ."
>>>>>
>>>>> I am probably not doing enough justice to the value of message
>>>>> IDs
>>>>> but,
>>>>> I will say that lots of very large companies use them in
>>>>> complex
>>>>> software applications to solve these and other issues.
>>>>>
>>>>> Cheers,
>>>>> Keith
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> vdsm-devel mailing list
>>>>> vdsm-devel(a)lists.fedorahosted.org
>>>>>
https://fedorahosted.org/mailman/listinfo/vdsm-devel
>>> _______________________________________________
>>> vdsm-devel mailing list
>>> vdsm-devel(a)lists.fedorahosted.org
>>>
https://fedorahosted.org/mailman/listinfo/vdsm-devel
>>>
>> _______________________________________________
>> vdsm-devel mailing list
>> vdsm-devel(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/vdsm-devel
>
_______________________________________________
vdsm-devel mailing list
vdsm-devel(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/vdsm-devel