On Mon, Apr 2, 2012 at 7:53 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, William Heinbockel wrote:
On Mon, Apr 2, 2012 at 5:37 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
I was under the impression from a previous thread that the structure would be used to encode namespaces. If we take this approach, I fail to see any real value in supporting structure...
Fields will (hopefully) come from 1 of 2 places:
- Lumberjack field library (predefined)
- Extension/app-specific field
We can either make this distinct in the field name or leave it up to the application.
I am in favour of leaving it up to the application, following an approach similar to yours, but I was under the impression the consensus swung the other way with the field names and namespaces.
If we don't have similar applications writing logs with similar structure, you may as well not bother structureing the logs as you will have to have different configurations for each thing, and at that point, individual parsers for each one is not significantly more work than dealing with different structure and element names for each one.
I don't think this makes a real difference from the processing side, but I will go either way. If we want to be explicit, then there has to be some indication as to what "namespace" the field names are from.
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
The cee will be updated to align with best practices coming out of this project.
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
I would rather not get caught up in these details
If we aren't dealing with these sorts of details, what are we trying to do?
I don't want to get muddled in the details of conversions, etc. I want this conversation to be focused on lumberjack. The CEE stuff will be updated based on the lumberjack.