Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
1. Core Fields -- flat names that are defined by lumberjack/CEE. These should encompass the minimal set of fields that are commonly used.
2. Extended Fields -- additional fields are either tied to a specific product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
Regards, Botond
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
David Lang
On Mon, Apr 2, 2012 at 5:37 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
I was under the impression from a previous thread that the structure would be used to encode namespaces. If we take this approach, I fail to see any real value in supporting structure...
Fields will (hopefully) come from 1 of 2 places:
1. Lumberjack field library (predefined) 2. Extension/app-specific field
We can either make this distinct in the field name or leave it up to the application.
I am in favour of leaving it up to the application, following an approach similar to yours, but I was under the impression the consensus swung the other way with the field names and namespaces.
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
The cee will be updated to align with best practices coming out of this project.
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
I would rather not get caught up in these details
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
David Lang
On Mon, 2 Apr 2012, William Heinbockel wrote:
On Mon, Apr 2, 2012 at 5:37 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
I was under the impression from a previous thread that the structure would be used to encode namespaces. If we take this approach, I fail to see any real value in supporting structure...
Fields will (hopefully) come from 1 of 2 places:
- Lumberjack field library (predefined)
- Extension/app-specific field
We can either make this distinct in the field name or leave it up to the application.
I am in favour of leaving it up to the application, following an approach similar to yours, but I was under the impression the consensus swung the other way with the field names and namespaces.
If we don't have similar applications writing logs with similar structure, you may as well not bother structureing the logs as you will have to have different configurations for each thing, and at that point, individual parsers for each one is not significantly more work than dealing with different structure and element names for each one.
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
The cee will be updated to align with best practices coming out of this project.
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
I would rather not get caught up in these details
If we aren't dealing with these sorts of details, what are we trying to do?
David Lang
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
David Lang
lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
On Mon, Apr 2, 2012 at 7:53 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, William Heinbockel wrote:
On Mon, Apr 2, 2012 at 5:37 PM, david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
I was under the impression from a previous thread that the structure would be used to encode namespaces. If we take this approach, I fail to see any real value in supporting structure...
Fields will (hopefully) come from 1 of 2 places:
- Lumberjack field library (predefined)
- Extension/app-specific field
We can either make this distinct in the field name or leave it up to the application.
I am in favour of leaving it up to the application, following an approach similar to yours, but I was under the impression the consensus swung the other way with the field names and namespaces.
If we don't have similar applications writing logs with similar structure, you may as well not bother structureing the logs as you will have to have different configurations for each thing, and at that point, individual parsers for each one is not significantly more work than dealing with different structure and element names for each one.
I don't think this makes a real difference from the processing side, but I will go either way. If we want to be explicit, then there has to be some indication as to what "namespace" the field names are from.
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
The cee will be updated to align with best practices coming out of this project.
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
I would rather not get caught up in these details
If we aren't dealing with these sorts of details, what are we trying to do?
I don't want to get muddled in the details of conversions, etc. I want this conversation to be focused on lumberjack. The CEE stuff will be updated based on the lumberjack.
On Mon, 2 Apr 2012 14:37:20 -0700 (PDT) david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
Ok, this can work also. Though it is not SourceOS which will make the distinction because Syslog exists on various OSes for example. The SyslogSeverity would be then NativeSeverity and LogSourceName or similar.
Though I'm not really convinced that this is a good approach because there always are a couple other fields which are source/protocol specific. For example in syslog, there is also SyslogFacility. If we keep this as is, then it is confusing to have some fields with the source/protocol name incuded the field and yet others are named generically. Trying to map all these source/protocol specific fields to a generic name in our field list might not be the best idea because this will make it a lot harder for the user to identify and/or map back to the original format. Another reason for keeping SyslogSeverity is that this is easier to validate and can possibly have a custom type because the value space is known. I was trying to take a hybrid approach and map only the common fields to generic ones and leave the names of the rest (uncommon and very source/protocol specific) as is.
Regards, Botond
On Tue, Apr 3, 2012 at 5:08 AM, Botond Botyanszki boti@nxlog.org wrote:
On Mon, 2 Apr 2012 14:37:20 -0700 (PDT) david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
Ok, this can work also. Though it is not SourceOS which will make the distinction because Syslog exists on various OSes for example. The SyslogSeverity would be then NativeSeverity and LogSourceName or similar.
Though I'm not really convinced that this is a good approach because there always are a couple other fields which are source/protocol specific. For example in syslog, there is also SyslogFacility. If we keep this as is, then it is confusing to have some fields with the source/protocol name incuded the field and yet others are named generically. Trying to map all these source/protocol specific fields to a generic name in our field list might not be the best idea because this will make it a lot harder for the user to identify and/or map back to the original format. Another reason for keeping SyslogSeverity is that this is easier to validate and can possibly have a custom type because the value space is known. I was trying to take a hybrid approach and map only the common fields to generic ones and leave the names of the rest (uncommon and very source/protocol specific) as is.
I think this is the best approach. Make things generic when the concept can be generalized, use more specific names when it cannot.
That is what I was trying to get at with the syslog priority/facility discussion in the other thread. These values only make sense within the context of syslog, therefore they should be named as such.
What I am cautious of is going too far down these routes as a project; it is very easy to start adding the fields/data from all of the individual protocols. What we should focus on is the most commonly used fields -- users, files, src/dst, time, etc.
On Tue, 3 Apr 2012 10:39:08 -0400 William Heinbockel wheinbockel@gmail.com wrote:
What I am cautious of is going too far down these routes as a project; it is very easy to start adding the fields/data from all of the individual protocols. What we should focus on is the most commonly used fields -- users, files, src/dst, time, etc.
Until developers start emitting CEE/luberjack, the field list we come up with will be used to map existing fields in log protocols and formats to. So IMO what we should do is take the most common N apps/protocols/formats such as Syslog , Windows EventLog, auditing (on linux, solaris, hpux etc), Android, checkpoint LEA, you name it. Then list up the fields used by these. Using this list, create a CEE/limberjack field list to which we can map these. Having done N, the N+1th will be likely pretty trivial to add.
The theoretical approach, namely that we come up with a field list we think will be useful, will fail at the very first encounter trying to put it in practice and normalize an existing log format to. Then you fix it, and again it will fail at the second. Instead of going through these iterations (the list I put in the wiki is a result of such), we should try to get it right with the practical approach. Maybe it won't look nice, because the list will be likely long and some will call it ugly, but will work better in practice and this is what users are after.
Regards, Botond
On Tue, 3 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:37:20 -0700 (PDT) david@lang.hm wrote:
On Mon, 2 Apr 2012, Botond Botyanszki wrote:
On Mon, 2 Apr 2012 14:34:22 -0400 William Heinbockel wheinbockel@gmail.com wrote:
Before we dive too deep into which fields to support and how to name them, we should first discuss the structure and organization of the names.
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
The functional product categories should be related to an organizational entity -- lumberjack, cee, IANA, etc.
We need some way to distinguish vendor/product namespaces from organization/functional ones.
If you specify the name of the product in the field name then there is no standardization. The point is that you should be able to handle audit subsystem events generically. If we have {"red_hat|auditd|filename":"..."} and {"solaris|auditing|filename":"..."}, then it is really hard to use the "filename" field across all product/application groups. Also one event would never contain both as in the following: {"red_hat|auditd|filename":"...", "solaris|auditing|filename":"..."} So I fail to see where this schema is useful.
Instead, the event should contain the product/app name in a separate field, i.e.: {"SourceOS":"solaris", "SourceName":"auditing", "filename":"..."} and {"SourceOS":"red-hat", "SourceName":"auditd", "filename":"..."}
Agreed. In a different thread I tried to make a similar point about the field definitions.
For example, we have the severity concept, but that will be different for things that natively think syslog than for things that nativly think window (and I understand the cee standard is different from either)
the severity tag needs to be the standards-defined tag, even if the conversion from native to cee is lossy
but there needs to also be a native_severity tag and SourceOS tag so that if something cares about the full details of the priginator, the data is available (but not in a Unix_severity, Windows_severity, etc proliferation of tags)
Ok, this can work also. Though it is not SourceOS which will make the distinction because Syslog exists on various OSes for example. The SyslogSeverity would be then NativeSeverity and LogSourceName or similar.
Though I'm not really convinced that this is a good approach because there always are a couple other fields which are source/protocol specific. For example in syslog, there is also SyslogFacility. If we keep this as is, then it is confusing to have some fields with the source/protocol name incuded the field and yet others are named generically. Trying to map all these source/protocol specific fields to a generic name in our field list might not be the best idea because this will make it a lot harder for the user to identify and/or map back to the original format. Another reason for keeping SyslogSeverity is that this is easier to validate and can possibly have a custom type because the value space is known. I was trying to take a hybrid approach and map only the common fields to generic ones and leave the names of the rest (uncommon and very source/protocol specific) as is.
I'm advocating something slightly different.
I'm saying that we must have generic names and values that we map to
but in addition to these generic names, we need to also store the raw native data.
Where we can, we should still have generic names for the native data, (so that we don't have syslog_severity, windows_severity, etc). It's not the field name that needs to be source specific, but the values in that field.
so you would have something like
cee: { native: { severity: value priority: value type: syslog } timestamp: value severity: value message: value processis: value userid: value processname: value }
David Lang
William Heinbockel wheinbockel@gmail.com writes:
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
Another contender to keep in mind is probably the journal - they have a few fields of their own (some of which the others, syslog-ng, rsyslog & nxlog could implement aswell, if so need be) that might worth being considered.
Lennart also expressed interest in emitting CEE-enhanced JSON from journal, as long as there's a clear algorithm to translate journal-internal names to CEE (but that's a future topic), and I'm fairly sure they'd be interested in the extended fields too.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
Definitely the former. I imagine most consumers will use dot-notation to reach into nested structures, and it is VERY unintuitive when you have "foo.bar.baz" and "foo|bar|baz.quux" too. (Which we will, since extensions are - I hope - allowed to nest)
We need some way to distinguish vendor/product namespaces from organization/functional ones.
Perhaps put vendor extensions under a "vendor" or "org" key?
{"org":{"redhat":{"foo":"bar"}}}
Perhaps a bit verbose, but it's simple and crystal clear, in my opinion.
On Tue, Apr 3, 2012 at 10:48 AM, Gergely Nagy algernon@balabit.hu wrote:
William Heinbockel wheinbockel@gmail.com writes:
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
Another contender to keep in mind is probably the journal - they have a few fields of their own (some of which the others, syslog-ng, rsyslog & nxlog could implement aswell, if so need be) that might worth being considered.
Lennart also expressed interest in emitting CEE-enhanced JSON from journal, as long as there's a clear algorithm to translate journal-internal names to CEE (but that's a future topic), and I'm fairly sure they'd be interested in the extended fields too.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
Definitely the former. I imagine most consumers will use dot-notation to reach into nested structures, and it is VERY unintuitive when you have "foo.bar.baz" and "foo|bar|baz.quux" too. (Which we will, since extensions are - I hope - allowed to nest)
We need some way to distinguish vendor/product namespaces from organization/functional ones.
Perhaps put vendor extensions under a "vendor" or "org" key?
{"org":{"redhat":{"foo":"bar"}}}
Perhaps a bit verbose, but it's simple and crystal clear, in my opinion.
This is the most intuitive to me and hopefully for most users. Though it appears that David and Botond are of the opposite mindset.
I guess there are two different but related conversations going on. The namespace of the fields and the producing application of the event record. In my opinion, we need both. The app can be pulled just like in the Syslog tag+pid. The namespace organization needs some thought.
On Tue, 3 Apr 2012, William Heinbockel wrote:
On Tue, Apr 3, 2012 at 10:48 AM, Gergely Nagy algernon@balabit.hu wrote:
William Heinbockel wheinbockel@gmail.com writes:
I propose we roughly follow the proposal from Balazs:
- Core Fields -- flat names that are defined by lumberjack/CEE. These
should encompass the minimal set of fields that are commonly used.
- Extended Fields -- additional fields are either tied to a specific
product/application or a functional product group (anti-virus, firewalls, etc.)
We should focus on scoping and defining #1, probably combining the libumberlog, patterndb, liblognorm, RFC5424/IANA fields. Maybe the best place to start is with the `auditd` list http://people.redhat.com/sgrubb/audit/audit-parse.txt and abstract linux-specific names to be more generally applicable.
Another contender to keep in mind is probably the journal - they have a few fields of their own (some of which the others, syslog-ng, rsyslog & nxlog could implement aswell, if so need be) that might worth being considered.
Lennart also expressed interest in emitting CEE-enhanced JSON from journal, as long as there's a clear algorithm to translate journal-internal names to CEE (but that's a future topic), and I'm fairly sure they'd be interested in the extended fields too.
For #2, we need to figure out how these work. I vote that the structure should infer the namespace and the namespace should either be unique to the vendor/product or developed by the community. {"red_hat":{"auditd":{"time":"..."}}} would work, or collapsed to {"red_hat|auditd|time":"..."} (feel free to substitute divisive char: !/.:|)
Definitely the former. I imagine most consumers will use dot-notation to reach into nested structures, and it is VERY unintuitive when you have "foo.bar.baz" and "foo|bar|baz.quux" too. (Which we will, since extensions are - I hope - allowed to nest)
We need some way to distinguish vendor/product namespaces from organization/functional ones.
Perhaps put vendor extensions under a "vendor" or "org" key?
{"org":{"redhat":{"foo":"bar"}}}
Perhaps a bit verbose, but it's simple and crystal clear, in my opinion.
This is the most intuitive to me and hopefully for most users. Though it appears that David and Botond are of the opposite mindset.
having a different namespace for each vendor makes it very hard to work with the logs across vendors.
I think there needs to be a tag to identify the vendor, but the generic tags that you will normally be working with should not be something that you have to know the vendor to use.
David Lang
I guess there are two different but related conversations going on. The namespace of the fields and the producing application of the event record. In my opinion, we need both. The app can be pulled just like in the Syslog tag+pid. The namespace organization needs some thought. _______________________________________________ lumberjack-developers mailing list lumberjack-developers@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/lumberjack-developers
lumberjack-developers@lists.fedorahosted.org