Opentracing and Kafka Propagation?

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Opentracing and Kafka Propagation?

Gerald Nunn
I'm using camel 2.23 with Spring Boot for a set of microservices that send
and receive messages via Kafka. I have the individual services working with
Jaegar with no issue however I was expecting transactions to span Kafka in
Jaeger and I'm not seeing that. Rather in Jaeger the transactions stop and
start with Kafka.

In my pom.xml file (
https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml) I'm
importing the following dependencies for tracing:

    <!-- Tracing -->
    <dependency>
      <groupId>org.apache.camel</groupId>
      <artifactId>camel-opentracing-starter</artifactId>
    </dependency>
    <dependency>
      <groupId>io.jaegertracing</groupId>
      <artifactId>jaeger-client</artifactId>
      <version>0.32.0</version>
    </dependency>
    <dependency>
      <groupId>io.jaegertracing</groupId>
      <artifactId>jaeger-tracerresolver</artifactId>
      <version>0.32.0</version>
    </dependency>
  </dependencies>

My application has the @CamelOpenTracing annotation on the Application
class. Finally I'm connecting to Kafka in Spring DSL as follows:

<route id="request-seat" routePolicyRef="policy">
...
<to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
 </route>

My understanding is that camel should propagate the tracing context
automatically but maybe I'm missing something? I see there is a
KafkaSpanDecorator in camel but it's not clear if this is an internal class
or something I should be using explicitly.

Thanks,

Gerald
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing and Kafka Propagation?

Gerald Nunn
So I am still stuck on this but some additional information. I enabled
trace logging in camel-opentracing and it looks like the appropriate code
is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
debug logs so I do not see anything from that.

Producer Logging:

21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start server
span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.587 [XNIO-1 task-1] INFO  request-seat - Starting route
21:33:40.601 [XNIO-1 task-1] INFO  request-seat -
com.redhat.demo.Registration@27bbd37
21:33:40.647 [XNIO-1 task-1] INFO  request-seat -
{"candidatename":"test","email":"[hidden email]
","seatname":"724","seatno":"19_22"}
21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client
span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.706 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.708 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO  i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
21:33:40.710 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO  i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO  i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
TRACE o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.711 [Camel (Registration) thread #1 - KafkaProducer[registration]]
INFO  i.j.i.reporters.LoggingReporter - Span reported:
ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
21:33:40.712 [XNIO-1 task-1] INFO  i.j.i.reporters.LoggingReporter - Span
reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start

Consumer Logging:

21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
","seatname":"724","seatno":"19_22"}
21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
","seatname":"724","seatno":"19_22"}
21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 i.j.i.reporters.LoggingReporter - Span reported:
bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 i.j.i.reporters.LoggingReporter - Span reported:
bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 i.j.i.reporters.LoggingReporter - Span reported:
1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
 i.j.i.reporters.LoggingReporter - Span reported:
72376bd92d1eee37:72376bd92d1eee37:0:1 - registration

On the Jaeger fight it shows the following detals for the producer in the
trace when you drill in:

camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
component: "camel-kafka"
internal.span.format: "jaeger"
message_bus.destination: "registration"
span.kind: "producer"

And on the consumer side I see:

camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
component: "camel-kafka"
internal.span.format: "jaeger"
kafka.offset: "909"
kafka.partition: "0"
message_bus.destination: "registration"
sampler.param: true
sampler.type: "const"
span.kind: "consumer"

Any suggestions on how to debug this would be appreciated.

Cheers,

Gerald

On Sat, 2 May 2020 at 22:53, Gerald Nunn <[hidden email]> wrote:

> I'm using camel 2.23 with Spring Boot for a set of microservices that send
> and receive messages via Kafka. I have the individual services working with
> Jaegar with no issue however I was expecting transactions to span Kafka in
> Jaeger and I'm not seeing that. Rather in Jaeger the transactions stop and
> start with Kafka.
>
> In my pom.xml file (
> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
> I'm importing the following dependencies for tracing:
>
>     <!-- Tracing -->
>     <dependency>
>       <groupId>org.apache.camel</groupId>
>       <artifactId>camel-opentracing-starter</artifactId>
>     </dependency>
>     <dependency>
>       <groupId>io.jaegertracing</groupId>
>       <artifactId>jaeger-client</artifactId>
>       <version>0.32.0</version>
>     </dependency>
>     <dependency>
>       <groupId>io.jaegertracing</groupId>
>       <artifactId>jaeger-tracerresolver</artifactId>
>       <version>0.32.0</version>
>     </dependency>
>   </dependencies>
>
> My application has the @CamelOpenTracing annotation on the Application
> class. Finally I'm connecting to Kafka in Spring DSL as follows:
>
> <route id="request-seat" routePolicyRef="policy">
> ...
> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
>  </route>
>
> My understanding is that camel should propagate the tracing context
> automatically but maybe I'm missing something? I see there is a
> KafkaSpanDecorator in camel but it's not clear if this is an internal class
> or something I should be using explicitly.
>
> Thanks,
>
> Gerald
>
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing and Kafka Propagation?

Gerald Nunn
So thanks to some help from Gary Brown we realized the issue is that
headers in Kafka are byte arrays and not strings. I've opened a JIRA issue
for it here:

https://issues.apache.org/jira/browse/CAMEL-15022

Cheers,

Gerald

On Sun, 3 May 2020 at 15:06, Gerald Nunn <[hidden email]> wrote:

> So I am still stuck on this but some additional information. I enabled
> trace logging in camel-opentracing and it looks like the appropriate code
> is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
> debug logs so I do not see anything from that.
>
> Producer Logging:
>
> 21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> 21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start server
> span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.587 [XNIO-1 task-1] INFO  request-seat - Starting route
> 21:33:40.601 [XNIO-1 task-1] INFO  request-seat -
> com.redhat.demo.Registration@27bbd37
> 21:33:40.647 [XNIO-1 task-1] INFO  request-seat -
> {"candidatename":"test","email":"[hidden email]
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client
> span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> 21:33:40.706 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client
> span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> 21:33:40.708 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 -
> registration
> 21:33:40.710 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: finish server
> span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.711 [Camel (Registration) thread #1 -
> KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> 21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> 21:33:40.712 [XNIO-1 task-1] INFO  i.j.i.reporters.LoggingReporter - Span
> reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
>
> Consumer Logging:
>
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
> ","seatname":"724","seatno":"19_22"}
> 21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  i.j.i.reporters.LoggingReporter - Span reported:
> bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  i.j.i.reporters.LoggingReporter - Span reported:
> bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  i.j.i.reporters.LoggingReporter - Span reported:
> 1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
>  i.j.i.reporters.LoggingReporter - Span reported:
> 72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
>
> On the Jaeger fight it shows the following detals for the producer in the
> trace when you drill in:
>
>
> camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> component: "camel-kafka"
> internal.span.format: "jaeger"
> message_bus.destination: "registration"
> span.kind: "producer"
>
> And on the consumer side I see:
>
>
> camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> component: "camel-kafka"
> internal.span.format: "jaeger"
> kafka.offset: "909"
> kafka.partition: "0"
> message_bus.destination: "registration"
> sampler.param: true
> sampler.type: "const"
> span.kind: "consumer"
>
> Any suggestions on how to debug this would be appreciated.
>
> Cheers,
>
> Gerald
>
> On Sat, 2 May 2020 at 22:53, Gerald Nunn <[hidden email]> wrote:
>
>> I'm using camel 2.23 with Spring Boot for a set of microservices that
>> send and receive messages via Kafka. I have the individual services working
>> with Jaegar with no issue however I was expecting transactions to span
>> Kafka in Jaeger and I'm not seeing that. Rather in Jaeger the transactions
>> stop and start with Kafka.
>>
>> In my pom.xml file (
>> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
>> I'm importing the following dependencies for tracing:
>>
>>     <!-- Tracing -->
>>     <dependency>
>>       <groupId>org.apache.camel</groupId>
>>       <artifactId>camel-opentracing-starter</artifactId>
>>     </dependency>
>>     <dependency>
>>       <groupId>io.jaegertracing</groupId>
>>       <artifactId>jaeger-client</artifactId>
>>       <version>0.32.0</version>
>>     </dependency>
>>     <dependency>
>>       <groupId>io.jaegertracing</groupId>
>>       <artifactId>jaeger-tracerresolver</artifactId>
>>       <version>0.32.0</version>
>>     </dependency>
>>   </dependencies>
>>
>> My application has the @CamelOpenTracing annotation on the Application
>> class. Finally I'm connecting to Kafka in Spring DSL as follows:
>>
>> <route id="request-seat" routePolicyRef="policy">
>> ...
>> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
>>  </route>
>>
>> My understanding is that camel should propagate the tracing context
>> automatically but maybe I'm missing something? I see there is a
>> KafkaSpanDecorator in camel but it's not clear if this is an internal class
>> or something I should be using explicitly.
>>
>> Thanks,
>>
>> Gerald
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Opentracing and Kafka Propagation?

Claus Ibsen-2
Hi

Thanks for reporting back here too. Yeah this would be great to get
resolved so we can have nice tracing from kafka.
Maybe Gary can help you with providing a fix for us - he has developed
the opentracing component AFAIR

On Wed, May 6, 2020 at 7:46 PM Gerald Nunn <[hidden email]> wrote:

>
> So thanks to some help from Gary Brown we realized the issue is that
> headers in Kafka are byte arrays and not strings. I've opened a JIRA issue
> for it here:
>
> https://issues.apache.org/jira/browse/CAMEL-15022
>
> Cheers,
>
> Gerald
>
> On Sun, 3 May 2020 at 15:06, Gerald Nunn <[hidden email]> wrote:
>
> > So I am still stuck on this but some additional information. I enabled
> > trace logging in camel-opentracing and it looks like the appropriate code
> > is being hit. Unfortunately KafkaSpanDecorator doesn't have any trace or
> > debug logs so I do not see anything from that.
> >
> > Producer Logging:
> >
> > 21:33:40.575 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> > 21:33:40.582 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.583 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start server
> > span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.587 [XNIO-1 task-1] INFO  request-seat - Starting route
> > 21:33:40.601 [XNIO-1 task-1] INFO  request-seat -
> > com.redhat.demo.Registration@27bbd37
> > 21:33:40.647 [XNIO-1 task-1] INFO  request-seat -
> > {"candidatename":"test","email":"[hidden email]
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.648 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client
> > span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> > 21:33:40.706 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client
> > span=ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 - registration
> > 21:33:40.708 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:956f2e15a975c890:6f383e9c12436146:1 -
> > registration
> > 21:33:40.710 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:6f383e9c12436146:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:14c1e23c4c1880f2:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: finish server
> > span=ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.711 [Camel (Registration) thread #1 -
> > KafkaProducer[registration]] INFO  i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:8e46815c4c58ae79:ae0f9b9afb996d5d:1 - start
> > 21:33:40.712 [XNIO-1 task-1] TRACE o.a.c.opentracing.OpenTracingTracer -
> > OpenTracing: start client span=ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> > 21:33:40.712 [XNIO-1 task-1] INFO  i.j.i.reporters.LoggingReporter - Span
> > reported: ae0f9b9afb996d5d:ae0f9b9afb996d5d:0:1 - start
> >
> > Consumer Logging:
> >
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.722 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start server
> > span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.723 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.729 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  reserve-seat - >>> {"candidatename":"test","email":"[hidden email]
> > ","seatname":"724","seatno":"19_22"}
> > 21:33:40.730 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> > span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: start client
> > span=bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  i.j.i.reporters.LoggingReporter - Span reported:
> > bd53803febcc7f5b:c6bde704862dedac:bd53803febcc7f5b:1 - sql
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  i.j.i.reporters.LoggingReporter - Span reported:
> > bd53803febcc7f5b:bd53803febcc7f5b:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  i.j.i.reporters.LoggingReporter - Span reported:
> > 1aa4209a9b2d6a64:1aa4209a9b2d6a64:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] TRACE
> > o.a.c.opentracing.OpenTracingTracer - OpenTracing: finish server
> > span=72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> > 21:33:40.740 [Camel (camel) thread #1 - KafkaConsumer[registration]] INFO
> >  i.j.i.reporters.LoggingReporter - Span reported:
> > 72376bd92d1eee37:72376bd92d1eee37:0:1 - registration
> >
> > On the Jaeger fight it shows the following detals for the producer in the
> > trace when you drill in:
> >
> >
> > camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> > component: "camel-kafka"
> > internal.span.format: "jaeger"
> > message_bus.destination: "registration"
> > span.kind: "producer"
> >
> > And on the consumer side I see:
> >
> >
> > camel.uri: "kafka://registration?brokers=seating-kafka-bootstrap%3A9092&clientId=registration-ui"
> > component: "camel-kafka"
> > internal.span.format: "jaeger"
> > kafka.offset: "909"
> > kafka.partition: "0"
> > message_bus.destination: "registration"
> > sampler.param: true
> > sampler.type: "const"
> > span.kind: "consumer"
> >
> > Any suggestions on how to debug this would be appreciated.
> >
> > Cheers,
> >
> > Gerald
> >
> > On Sat, 2 May 2020 at 22:53, Gerald Nunn <[hidden email]> wrote:
> >
> >> I'm using camel 2.23 with Spring Boot for a set of microservices that
> >> send and receive messages via Kafka. I have the individual services working
> >> with Jaegar with no issue however I was expecting transactions to span
> >> Kafka in Jaeger and I'm not seeing that. Rather in Jaeger the transactions
> >> stop and start with Kafka.
> >>
> >> In my pom.xml file (
> >> https://github.com/gnunn1/seating/blob/master/registration-ui/pom.xml)
> >> I'm importing the following dependencies for tracing:
> >>
> >>     <!-- Tracing -->
> >>     <dependency>
> >>       <groupId>org.apache.camel</groupId>
> >>       <artifactId>camel-opentracing-starter</artifactId>
> >>     </dependency>
> >>     <dependency>
> >>       <groupId>io.jaegertracing</groupId>
> >>       <artifactId>jaeger-client</artifactId>
> >>       <version>0.32.0</version>
> >>     </dependency>
> >>     <dependency>
> >>       <groupId>io.jaegertracing</groupId>
> >>       <artifactId>jaeger-tracerresolver</artifactId>
> >>       <version>0.32.0</version>
> >>     </dependency>
> >>   </dependencies>
> >>
> >> My application has the @CamelOpenTracing annotation on the Application
> >> class. Finally I'm connecting to Kafka in Spring DSL as follows:
> >>
> >> <route id="request-seat" routePolicyRef="policy">
> >> ...
> >> <to id="send-seat" uri="kafka:registration?clientId=registration-ui"/>
> >>  </route>
> >>
> >> My understanding is that camel should propagate the tracing context
> >> automatically but maybe I'm missing something? I see there is a
> >> KafkaSpanDecorator in camel but it's not clear if this is an internal class
> >> or something I should be using explicitly.
> >>
> >> Thanks,
> >>
> >> Gerald
> >>
> >



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2