syslog RFC 5424 dataformat component

[geemang]

 The current syslog is using an outdated RFC 3164.  Since, I'm in need of a syslog RFC 5424 component I've created one that current wraps syslog4j for the marshalling, the rest I can pretty much copy and extend the existing syslog implementation.

Questions
1) I'm open to contributing this, do you guys want this if it uses syslog4j under the covers?
2) If so, what should it be called? syslog2?
 
Cheers,
Martin

[Willem.Jiang]

Hi,

I'm not sure if the RFC 5424 has back ward compatibility to RFC 3164.
If it is we could keep the component name with camel-syslog, otherwise
would move on to call it camel-syslog2.

I just have a quick look at the syslog4j license, it is using GPL which
means we cannot host this component in Apache repo. But we have the
camel-extra project[1] which can host this kind of components.

[1]https://code.google.com/a/apache-extras.org/p/camel-extra/

On Wed Nov 23 11:11:12 2011, geemang wrote:

>   The current syslog is using an outdated RFC 3164.  Since, I'm in need of a
> syslog RFC 5424 component I've created one that current wraps syslog4j for
> the marshalling, the rest I can pretty much copy and extend the existing
> syslog implementation.
>
> Questions
> 1) I'm open to contributing this, do you guys want this if it uses syslog4j
> under the covers?
> 2) If so, what should it be called? syslog2?
>
> Cheers,
> Martin
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5015386.html
> Sent from the Camel Development mailing list archive at Nabble.com.
>

--
Willem
----------------------------------
FuseSource
Web: http://www.fusesource.com
Blog:    http://willemjiang.blogspot.com (English)
          http://jnn.javaeye.com (Chinese)
Twitter: willemjiang
Weibo: willemjiang

[geemang]

The spec does not say anything about backwards compatibility, but I believe much of the syslog code in the existing component is reusable.  

ASSUMPTIONS: we handle TLS / SSL etc with the netty / mina layer. SYSLOG4J is being used right now, but likely optional. Writing and parsing RFC5424 is not that hard, you guys already handle most of it. So, time permitting, we might move on without it.

A RFC5424 syslog event starts like this: <110>1   where the '1' after the > denotes that this is a version greater than 3164. So it's possible to detect this when doing unmarshal and parse the message appropriately.

On the unmarshal side we're not so lucky, RFC5424 introduces STRUCTURED-DATA with is a defined structure for the message part. We'd need someway to tell the syslog that we'd like to write structured data.  

If we enhance the existing syslog component, it'd require some someway the user could select the appropriate version.  It'd be nice to handle both RFC's with one component.

Is there best practice you can suggest here?

[Johan Edstrom-2]

I'd be nice if we could just handle both.
I'm game if you want to chat about it, the first one I just wrote to get a basic setup, the whole syslog
world suffers from not being that greatly standardized :)

/je

On Nov 23, 2011, at 7:09 AM, geemang wrote:

> The spec does not say anything about backwards compatibility, but I believe
> much of the syslog code in the existing component is reusable.  
>
> ASSUMPTIONS: we handle TLS / SSL etc with the netty / mina layer. SYSLOG4J
> is being used right now, but likely optional. Writing and parsing RFC5424 is
> not that hard, you guys already handle most of it. So, time permitting, we
> might move on without it.
>
> A RFC5424 syslog event starts like this: <110>1   where the '1' after the >
> denotes that this is a version greater than 3164. So it's possible to detect
> this when doing unmarshal and parse the message appropriately.
>
> On the unmarshal side we're not so lucky, RFC5424 introduces STRUCTURED-DATA
> with is a defined structure for the message part. We'd need someway to tell
> the syslog that we'd like to write structured data.  
>
> If we enhance the existing syslog component, it'd require some someway the
> user could select the appropriate version.  It'd be nice to handle both
> RFC's with one component.
>
> Is there best practice you can suggest here?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5016770.html
> Sent from the Camel Development mailing list archive at Nabble.com.

[Johan Edstrom-2]

It doesn't look from just reading the RFC as if it would be that hard to distinguish

Between :

<34>Oct 11 22:14:15 mymachine su: 'su root' failed for lonvick on /dev/pts/8
<34>1 2003-10-11T22:14:15.003Z mymachine.example.com su - ID47 - BOM'su root' failed for lonvick on /dev/pts/8

The latter being 5424 format..

As well as :


<165>Aug 24 05:34:00 CST 1987 mymachine myproc[10]: %% It's
         time to make the do-nuts.  %%  Ingredients: Mix=OK, Jelly=OK #
         Devices: Mixer=OK, Jelly_Injector=OK, Frier=OK # Transport:
         Conveyer1=OK, Conveyer2=OK # %%

<165>1 2003-10-11T22:14:15.003Z mymachine.example.com
           evntslog - ID47 [exampleSDID@32473 iut="3" eventSource=
           "Application" eventID="1011"] BOMAn application
           event log entry…


I.e :

<165>1 looks like enough to at least make a guesstimate?

/je

On Nov 22, 2011, at 11:50 PM, Willem Jiang wrote:

> Hi,
>
> I'm not sure if the RFC 5424 has back ward compatibility to RFC 3164.
> If it is we could keep the component name with camel-syslog, otherwise would move on to call it camel-syslog2.
>
> I just have a quick look at the syslog4j license, it is using GPL which means we cannot host this component in Apache repo. But we have the camel-extra project[1] which can host this kind of components.
>
> [1]https://code.google.com/a/apache-extras.org/p/camel-extra/
>
> On Wed Nov 23 11:11:12 2011, geemang wrote:
>>  The current syslog is using an outdated RFC 3164.  Since, I'm in need of a
>> syslog RFC 5424 component I've created one that current wraps syslog4j for
>> the marshalling, the rest I can pretty much copy and extend the existing
>> syslog implementation.
>>
>> Questions
>> 1) I'm open to contributing this, do you guys want this if it uses syslog4j
>> under the covers?
>> 2) If so, what should it be called? syslog2?
>>
>> Cheers,
>> Martin
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5015386.html
>> Sent from the Camel Development mailing list archive at Nabble.com.
>>
>
>
> --
> Willem
> ----------------------------------
> FuseSource
> Web: http://www.fusesource.com
> Blog:    http://willemjiang.blogspot.com (English)
>         http://jnn.javaeye.com (Chinese)
> Twitter: willemjiang
> Weibo: willemjiang

[hadrian]

Yes, it should be fairly easy to do it in one component. The selection
between the two versions can be done either with a uri parameter, or a
different uri scheme (the former sounds like the better option).

Cheers,
Hadrian



On 11/23/2011 09:09 AM, geemang wrote:

> The spec does not say anything about backwards compatibility, but I believe
> much of the syslog code in the existing component is reusable.
>
> ASSUMPTIONS: we handle TLS / SSL etc with the netty / mina layer. SYSLOG4J
> is being used right now, but likely optional. Writing and parsing RFC5424 is
> not that hard, you guys already handle most of it. So, time permitting, we
> might move on without it.
>
> A RFC5424 syslog event starts like this:<110>1   where the '1' after the>
> denotes that this is a version greater than 3164. So it's possible to detect
> this when doing unmarshal and parse the message appropriately.
>
> On the unmarshal side we're not so lucky, RFC5424 introduces STRUCTURED-DATA
> with is a defined structure for the message part. We'd need someway to tell
> the syslog that we'd like to write structured data.
>
> If we enhance the existing syslog component, it'd require some someway the
> user could select the appropriate version.  It'd be nice to handle both
> RFC's with one component.
>
> Is there best practice you can suggest here?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5016770.html
> Sent from the Camel Development mailing list archive at Nabble.com.

--
Hadrian Zbarcea
Principal Software Architect
Talend, Inc
http://coders.talend.com/
http://camelbot.blogspot.com/

[geemang]

How would you like to proceed?

It'd also be nice to handle the structured data un/marshal with a map or something. Perhaps a property on the SyslogMessage object?

[Johan Edstrom-2]

I'll just need to get past the Turkey hibernation :)
But it would be awesome I think if we could remain in the ASF license land with one
component, checking for which converter to use in the converters and specifying manually
one of the two in a declared context?

That make sense?
So we'd need an unmarshaller (which looks the same as the other one more or less)
and a marshaller as well to spit out the message.

And yeah, adding to the Syslog object seems a good idea to me, I do not think there are a
ton of users of it right now but adding fields shouldn't be the end of the world.

Sound good?

Thanks!

On Nov 23, 2011, at 3:33 PM, geemang wrote:

> How would you like to proceed?
>
> It'd also be nice to handle the structured data un/marshal with a map or
> something. Perhaps a property on the SyslogMessage object?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5018448.html
> Sent from the Camel Development mailing list archive at Nabble.com.

[Claus Ibsen-2]

On Thu, Nov 24, 2011 at 9:55 PM, Johan Edstrom <[hidden email]> wrote:

> I'll just need to get past the Turkey hibernation :)
> But it would be awesome I think if we could remain in the ASF license land with one
> component, checking for which converter to use in the converters and specifying manually
> one of the two in a declared context?
>
> That make sense?
> So we'd need an unmarshaller (which looks the same as the other one more or less)
> and a marshaller as well to spit out the message.
>
> And yeah, adding to the Syslog object seems a good idea to me, I do not think there are a
> ton of users of it right now but adding fields shouldn't be the end of the world.
>
> Sound good?
>

I like this. +1 from here.

> Thanks!
>
> On Nov 23, 2011, at 3:33 PM, geemang wrote:
>
>> How would you like to proceed?
>>
>> It'd also be nice to handle the structured data un/marshal with a map or
>> something. Perhaps a property on the SyslogMessage object?
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/syslog-RFC-5424-dataformat-component-tp5015386p5018448.html
>> Sent from the Camel Development mailing list archive at Nabble.com.
>
>

--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

[geemang]

Sounds good to me.  I'm just above novice when it comes to Camel, but I've got some cycles, how do you want to proceed?   I have some working code as I hacked this together for a customer demo.

[geemang]

How'd you get around this issue:  https://issues.apache.org/jira/browse/CAMEL-3607   ??

By default it looks like netty uses receiveBuffer size of 768, I can't figure out how to configure the netty component to allow 1024 (which of course syslog should handle).

I see this in the logs
Body:BigEndianHeapChannelBuffer(ridx=0, widx=768, cap=768)


FYI: sending the 1024 syslog message over netty looks to work fine

[subramanyar]

Hi,

I am trying to migrate the syslog message formats from RFC 3164 to RFC 5424. I am having following issues
here.

1. I cannot use the syslog-ng since it is a GPL licensed.

Is there any package available commercially or apache licensed packaged which I can use in place of the syslog-ng.
 
Are there any no GPL syslog interceptors available which I can use & modify my message in the latest format.

-Subramanya