Logging integration entry and exit with breadcrumbs

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Logging integration entry and exit with breadcrumbs

Stig Døssing
Hi,

We'd like to log when our integrations start and finish processing of a message, along with an identifier for the message and derived exchanges. We'd prefer if this could be done globally for all integrations in some common configuration, rather than having to remind our developers to put log statements in at the start and end of routes that receive messages from outside Camel (e.g. via JMS or REST endpoints).

Basically we'd like to only log "at the edge" of Camel, i.e. when a message is picked up by Camel from an external source. We don't want to log exchanges that are created internally, as part of processing another exchange.

For instance given

From("jms:queue:someQueue")
  .routeId("jmsRoute")
  .enrich("direct:endpoint1")

From("direct:endpoint1")
  .routeId("helloRoute")
  .setBody("hello")

We would like to only log when a message starts going through "jmsRoute" as well as when such an exchange is completed, but not when an exchange is created for "helloRoute".

We've enabled breadcrumb ids to help us track which log messages belong to which message, and have implemented an EventNotifierSupport which receives ExchangeCreatedEvent and ExchangeCompletedEvent.

It seems to us that the ExchangeCreatedEvent is sent before the MDC with the breadcrumb is set, so our logging of ExchangeCreatedEvent does not contain the breadcrumb. We are also having a bit of trouble deciding how to filter out exchange events we don't want to log. We're currently looking at the exchange getFromRouteId, and the unit of work route context to filter out created/completed events.

Is there a better mechanism for achieving this than using EventNotifier and exchange events? Is it intended that ExchangeCreatedEvent is processed before the MDC is populated?
Reply | Threaded
Open this post in threaded view
|

Re: Logging integration entry and exit with breadcrumbs

Claus Ibsen-2
The created event is happening when the exchange is created and before
routing and unit of work is setup, and therefore also before mdc unit
of work. And yes its intended.

The breadcrumb id is just the exchange id, so you can just get and use that


On Thu, Sep 5, 2019 at 2:23 PM Stig Døssing <[hidden email]> wrote:

>
> Hi,
>
> We'd like to log when our integrations start and finish processing of a message, along with an identifier for the message and derived exchanges. We'd prefer if this could be done globally for all integrations in some common configuration, rather than having to remind our developers to put log statements in at the start and end of routes that receive messages from outside Camel (e.g. via JMS or REST endpoints).
>
> Basically we'd like to only log "at the edge" of Camel, i.e. when a message is picked up by Camel from an external source. We don't want to log exchanges that are created internally, as part of processing another exchange.
>
> For instance given
>
> From("jms:queue:someQueue")
>   .routeId("jmsRoute")
>   .enrich("direct:endpoint1")
>
> From("direct:endpoint1")
>   .routeId("helloRoute")
>   .setBody("hello")
>
> We would like to only log when a message starts going through "jmsRoute" as well as when such an exchange is completed, but not when an exchange is created for "helloRoute".
>
> We've enabled breadcrumb ids to help us track which log messages belong to which message, and have implemented an EventNotifierSupport which receives ExchangeCreatedEvent and ExchangeCompletedEvent.
>
> It seems to us that the ExchangeCreatedEvent is sent before the MDC with the breadcrumb is set, so our logging of ExchangeCreatedEvent does not contain the breadcrumb. We are also having a bit of trouble deciding how to filter out exchange events we don't want to log. We're currently looking at the exchange getFromRouteId, and the unit of work route context to filter out created/completed events.
>
> Is there a better mechanism for achieving this than using EventNotifier and exchange events? Is it intended that ExchangeCreatedEvent is processed before the MDC is populated?



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

RE: Logging integration entry and exit with breadcrumbs

Stig Døssing
Thanks for the quick response.

We sometimes receive messages with the breadcrumb id already set, so we can track messages across the entire system and not just inside Camel, so printing the exchange id would be misleading. Maybe we can extract the breadcrumb id from the exchange in the notifier manually.

Is there a better way to only log when a message crosses the boundary in/out of Camel than to use EventNotifier and ExchangeCreated/CompletedEvent? Is it reasonable to look at the exchange getFromRouteId and unit of work route context (we check these for null) to decide whether the exchange was created due to a message from outside Camel, or is there another way to decide this?



-----Original Message-----
From: Claus Ibsen <[hidden email]>
Sent: 5. september 2019 14:43
To: [hidden email]
Subject: Re: Logging integration entry and exit with breadcrumbs

The created event is happening when the exchange is created and before routing and unit of work is setup, and therefore also before mdc unit of work. And yes its intended.

The breadcrumb id is just the exchange id, so you can just get and use that


On Thu, Sep 5, 2019 at 2:23 PM Stig Døssing <[hidden email]> wrote:

>
> Hi,
>
> We'd like to log when our integrations start and finish processing of a message, along with an identifier for the message and derived exchanges. We'd prefer if this could be done globally for all integrations in some common configuration, rather than having to remind our developers to put log statements in at the start and end of routes that receive messages from outside Camel (e.g. via JMS or REST endpoints).
>
> Basically we'd like to only log "at the edge" of Camel, i.e. when a message is picked up by Camel from an external source. We don't want to log exchanges that are created internally, as part of processing another exchange.
>
> For instance given
>
> From("jms:queue:someQueue")
>   .routeId("jmsRoute")
>   .enrich("direct:endpoint1")
>
> From("direct:endpoint1")
>   .routeId("helloRoute")
>   .setBody("hello")
>
> We would like to only log when a message starts going through "jmsRoute" as well as when such an exchange is completed, but not when an exchange is created for "helloRoute".
>
> We've enabled breadcrumb ids to help us track which log messages belong to which message, and have implemented an EventNotifierSupport which receives ExchangeCreatedEvent and ExchangeCompletedEvent.
>
> It seems to us that the ExchangeCreatedEvent is sent before the MDC with the breadcrumb is set, so our logging of ExchangeCreatedEvent does not contain the breadcrumb. We are also having a bit of trouble deciding how to filter out exchange events we don't want to log. We're currently looking at the exchange getFromRouteId, and the unit of work route context to filter out created/completed events.
>
> Is there a better mechanism for achieving this than using EventNotifier and exchange events? Is it intended that ExchangeCreatedEvent is processed before the MDC is populated?



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Logging integration entry and exit with breadcrumbs

Claus Ibsen-2
Yeah just look for that existing breadcrumb in the header

There are sending events too, when Camel sends it to endpoints (can be
internal or external depending on component - eg direct vs http).
An exchange is always created by Camel as its a Camel holder.
The message on the other hand is the holder of the payload and
metadata (eg headers).

So by created outside Camel I assume its some external system that
sends a message to Camel eg via JMS or HTTP. As opposed to a Camel
timer that is created internally by Camel.
And no you would have to see this per component how they "operate".


On Fri, Sep 6, 2019 at 10:16 AM Stig Døssing
<[hidden email]> wrote:

>
> Thanks for the quick response.
>
> We sometimes receive messages with the breadcrumb id already set, so we can track messages across the entire system and not just inside Camel, so printing the exchange id would be misleading. Maybe we can extract the breadcrumb id from the exchange in the notifier manually.
>
> Is there a better way to only log when a message crosses the boundary in/out of Camel than to use EventNotifier and ExchangeCreated/CompletedEvent? Is it reasonable to look at the exchange getFromRouteId and unit of work route context (we check these for null) to decide whether the exchange was created due to a message from outside Camel, or is there another way to decide this?
>
>
>
> -----Original Message-----
> From: Claus Ibsen <[hidden email]>
> Sent: 5. september 2019 14:43
> To: [hidden email]
> Subject: Re: Logging integration entry and exit with breadcrumbs
>
> The created event is happening when the exchange is created and before routing and unit of work is setup, and therefore also before mdc unit of work. And yes its intended.
>
> The breadcrumb id is just the exchange id, so you can just get and use that
>
>
> On Thu, Sep 5, 2019 at 2:23 PM Stig Døssing <[hidden email]> wrote:
> >
> > Hi,
> >
> > We'd like to log when our integrations start and finish processing of a message, along with an identifier for the message and derived exchanges. We'd prefer if this could be done globally for all integrations in some common configuration, rather than having to remind our developers to put log statements in at the start and end of routes that receive messages from outside Camel (e.g. via JMS or REST endpoints).
> >
> > Basically we'd like to only log "at the edge" of Camel, i.e. when a message is picked up by Camel from an external source. We don't want to log exchanges that are created internally, as part of processing another exchange.
> >
> > For instance given
> >
> > From("jms:queue:someQueue")
> >   .routeId("jmsRoute")
> >   .enrich("direct:endpoint1")
> >
> > From("direct:endpoint1")
> >   .routeId("helloRoute")
> >   .setBody("hello")
> >
> > We would like to only log when a message starts going through "jmsRoute" as well as when such an exchange is completed, but not when an exchange is created for "helloRoute".
> >
> > We've enabled breadcrumb ids to help us track which log messages belong to which message, and have implemented an EventNotifierSupport which receives ExchangeCreatedEvent and ExchangeCompletedEvent.
> >
> > It seems to us that the ExchangeCreatedEvent is sent before the MDC with the breadcrumb is set, so our logging of ExchangeCreatedEvent does not contain the breadcrumb. We are also having a bit of trouble deciding how to filter out exchange events we don't want to log. We're currently looking at the exchange getFromRouteId, and the unit of work route context to filter out created/completed events.
> >
> > Is there a better mechanism for achieving this than using EventNotifier and exchange events? Is it intended that ExchangeCreatedEvent is processed before the MDC is populated?
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

RE: Logging integration entry and exit with breadcrumbs

Stig Døssing
Yes, that's what I mean by "created outside Camel". Some external system sending a message into Camel.

Okay thanks, we'll see if we can figure out a method for deciding when to log.

-----Original Message-----
From: Claus Ibsen <[hidden email]>
Sent: 6. september 2019 10:31
To: [hidden email]
Subject: Re: Logging integration entry and exit with breadcrumbs

Yeah just look for that existing breadcrumb in the header

There are sending events too, when Camel sends it to endpoints (can be internal or external depending on component - eg direct vs http).
An exchange is always created by Camel as its a Camel holder.
The message on the other hand is the holder of the payload and metadata (eg headers).

So by created outside Camel I assume its some external system that sends a message to Camel eg via JMS or HTTP. As opposed to a Camel timer that is created internally by Camel.
And no you would have to see this per component how they "operate".


On Fri, Sep 6, 2019 at 10:16 AM Stig Døssing <[hidden email]> wrote:

>
> Thanks for the quick response.
>
> We sometimes receive messages with the breadcrumb id already set, so we can track messages across the entire system and not just inside Camel, so printing the exchange id would be misleading. Maybe we can extract the breadcrumb id from the exchange in the notifier manually.
>
> Is there a better way to only log when a message crosses the boundary in/out of Camel than to use EventNotifier and ExchangeCreated/CompletedEvent? Is it reasonable to look at the exchange getFromRouteId and unit of work route context (we check these for null) to decide whether the exchange was created due to a message from outside Camel, or is there another way to decide this?
>
>
>
> -----Original Message-----
> From: Claus Ibsen <[hidden email]>
> Sent: 5. september 2019 14:43
> To: [hidden email]
> Subject: Re: Logging integration entry and exit with breadcrumbs
>
> The created event is happening when the exchange is created and before routing and unit of work is setup, and therefore also before mdc unit of work. And yes its intended.
>
> The breadcrumb id is just the exchange id, so you can just get and use
> that
>
>
> On Thu, Sep 5, 2019 at 2:23 PM Stig Døssing <[hidden email]> wrote:
> >
> > Hi,
> >
> > We'd like to log when our integrations start and finish processing of a message, along with an identifier for the message and derived exchanges. We'd prefer if this could be done globally for all integrations in some common configuration, rather than having to remind our developers to put log statements in at the start and end of routes that receive messages from outside Camel (e.g. via JMS or REST endpoints).
> >
> > Basically we'd like to only log "at the edge" of Camel, i.e. when a message is picked up by Camel from an external source. We don't want to log exchanges that are created internally, as part of processing another exchange.
> >
> > For instance given
> >
> > From("jms:queue:someQueue")
> >   .routeId("jmsRoute")
> >   .enrich("direct:endpoint1")
> >
> > From("direct:endpoint1")
> >   .routeId("helloRoute")
> >   .setBody("hello")
> >
> > We would like to only log when a message starts going through "jmsRoute" as well as when such an exchange is completed, but not when an exchange is created for "helloRoute".
> >
> > We've enabled breadcrumb ids to help us track which log messages belong to which message, and have implemented an EventNotifierSupport which receives ExchangeCreatedEvent and ExchangeCompletedEvent.
> >
> > It seems to us that the ExchangeCreatedEvent is sent before the MDC with the breadcrumb is set, so our logging of ExchangeCreatedEvent does not contain the breadcrumb. We are also having a bit of trouble deciding how to filter out exchange events we don't want to log. We're currently looking at the exchange getFromRouteId, and the unit of work route context to filter out created/completed events.
> >
> > Is there a better mechanism for achieving this than using EventNotifier and exchange events? Is it intended that ExchangeCreatedEvent is processed before the MDC is populated?
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2