Camel Tracer and Logstash

classic Classic list List threaded Threaded
2 messages Options
Reply | Threaded
Open this post in threaded view
|

Camel Tracer and Logstash

Karts
Has anyone managed to integrate Camel logs with Logstash/Elasticsearch?

I've got tracer and MDC logging enabled in my CamelContext and am sending logs to Logstash via log4j TCP appender. This is working great (especially MDC fields), but I'm trying to further parse and filter the message to individual fields in Elasticsearch.

For example, my message is:
ID-dx123abc001-abc-com-42264-1475782902034-0-193168 >>> (system-sql) setHeader[Content-Type, application/json] --> marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55] <<< Pattern:InOut, Headers:{Accept-Encoding=gzip, deflate, Authorization=Basic aaaaXRpeDpreG1pZGFz, breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169, CamelHttpMethod=GET, CamelHttpPath=, CamelHttpQuery=name=new-strategies, CamelHttpServletRequest=Request(GET /esb/system/sql?name=new-strategies)@6aaa8b00, CamelHttpServletResponse=HTTP/1.1 200 , CamelHttpUri=/esb/system/sql, CamelHttpUrl=http://system.abc.com:51950/esb/system/sql, CamelJdbcColumnNames=[STRATEGY_NAME], CamelJdbcRowCount=2, CamelServletContextPath=/esb/system/sql, Connection=keep-alive, Content-Type=application/json, Host=system.abc.com:51950, name=new-strategies, User-Agent=Apache-HttpClient/4.2.1 (java 1.5)}, BodyType:java.util.ArrayList, Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2 }]

And I would like logstash to parse and filter this so that it is stored in Elasticsearch as individual fields:
...
from=setHeader[Content-Type, application/json]
to=marshal[org.apache.camel.component.jackson.JacksonDataFormat@7e38f55]
breadcrumbId=ID-dx123abc001-abc-com-42264-1475782902034-0-193169
CamelJdbcColumnNames=[STRATEGY_NAME]
CamelJdbcRowCount=2
BodyType:java.util.ArrayList
Body:[{STRATEGY_NAME=S1 }, {STRATEGY_NAME=S2 }]
etc.

Has anyone been able to do this? I'm struggling with getting this to work with grok or ruby in the logstash config. Any tips are appreciated!

Thanks
Reply | Threaded
Open this post in threaded view
|

Re: Camel Tracer and Logstash

Karts
In case anyone needs it, we used grok to achieve this:

if [path] =~ "Tracer" {
        grok {
                patterns_dir => ["./extra_patterns"]
                match => {
                        "message" => "^\n%{USERNAME:id} >>> \(%{USERNAME:routeId}\) %{GREEDYDATA:source} --> %{GREEDYDATA:destination} <<<\n, Pattern:%{WORD:pattern}\n, Headers:{%{MULTILINEGREEDYDATA:headers}}\n, BodyType:%{JAVAFILE:bodyType}\n, Body:%{GREEDYDATA:body}"
                }
        }
        kv {
                source => "headers"
                field_split => ","
        }
  }