Help Needed: Splitter and Parallel Processing

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Help Needed: Splitter and Parallel Processing

Luiz Eduardo Valmont
Hi, friends!

According to instructions found on this link (splitter)
<https://camel.apache.org/splitter.html> and this other link (bean)
<https://camel.apache.org/bean.html>, I wrote this (simple?) route:

  *public* *void* configure() *throws* Exception {

>     from( "quartz2://instancesProcessor?cron=" + quartz )
>       .routeId( "InstancesProcessor" )
>         .process( loadUnprocessedInstancesProcessor )
>         .split()
>           .body()
>           .log( LoggingLevel.WARN, "Body type is:
> ${body.getClass().getName()}" )
>           .bean( instanceHandler, "handle" )
>           .log( LoggingLevel.WARN, "Handled one of:
> ${body.getClass().getName()}" )
>         .end()
>       .end()
>     .end()
>     ;
>

And this is the general output:

2019-08-06 04:04:20.281 TRACE fcdc964e5ee8 --- [aultQuartzScheduler-Orthanc

> Overseer_Worker-1] .r.LoadUnprocessedInstancesProcessorImpl : Found 33
> unprocessed instances
> 2019-08-06 04:04:20.373  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.507  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.518 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.559 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.608  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: model.Instance
> 2019-08-06 04:04:20.610  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.617  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.621 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.625 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.750  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: [..]
> 2019-08-06 04:04:20.751  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:30.017 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed instances
> 2019-08-06 04:04:30.020  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:40.020 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed instances
> 2019-08-06 04:04:40.027  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3] InstancesProcessor
>               : Body type is: model.Instance
>

It basically process 2 instances out of 33 available. And that's it. For
the following interations, it merely logs a single Body type. Since it
executes at least once, I reckon that "type-wise" I'm ok, else nothing
would work. I suppose I'm running out of consumers because of some other
route. But it still beats me because, as the logging shows, the
instanceHandler bean does not hang.

Question: what can be wrong/faulty in my routing setup? I wanted to execute
the instanceHandler in parallel, like X at a time. But one step at a time.

Any help or guidance or guess are mostly welcome. Even wild guesses.

Thanks in advance!

Luiz Eduardo Guida Valmont

(21) 2220-6223
(21) 98663-1337

ALIS - Soluções em Engenharia & Sistemas
Reply | Threaded
Open this post in threaded view
|

AW: Help Needed: Splitter and Parallel Processing

Burkard Stephan
Hi

OK, here is my wild guess :-)


Based on your log output I suspect the following:

1. It starts and finds 33 instances to process
2. After processing the second instance, there is a problem
3. Could be that the log output "Handled one of: [..]" instead of "Handled one of: model.Instance" is an indicator for that.
4. Processing is aborted, perhaps the instanceHandler is swallowing exceptions
5. The next trigger starts, it still finds 33 instances because the already processed ones are not removed from the list of unprocessed
6. It tries to process the first, but fails because it is already processed.
7. Processing is aborted
8. Endless loop between #5 and #7

Cheers
Stephan



-----Ursprüngliche Nachricht-----
Von: Luiz Eduardo Valmont <[hidden email]>
Gesendet: Dienstag, 6. August 2019 06:25
An: [hidden email]
Betreff: Help Needed: Splitter and Parallel Processing

Hi, friends!

According to instructions found on this link (splitter) <https://camel.apache.org/splitter.html> and this other link (bean) <https://camel.apache.org/bean.html>, I wrote this (simple?) route:

  *public* *void* configure() *throws* Exception {

>     from( "quartz2://instancesProcessor?cron=" + quartz )
>       .routeId( "InstancesProcessor" )
>         .process( loadUnprocessedInstancesProcessor )
>         .split()
>           .body()
>           .log( LoggingLevel.WARN, "Body type is:
> ${body.getClass().getName()}" )
>           .bean( instanceHandler, "handle" )
>           .log( LoggingLevel.WARN, "Handled one of:
> ${body.getClass().getName()}" )
>         .end()
>       .end()
>     .end()
>     ;
>

And this is the general output:

2019-08-06 04:04:20.281 TRACE fcdc964e5ee8 --- [aultQuartzScheduler-Orthanc

> Overseer_Worker-1] .r.LoadUnprocessedInstancesProcessorImpl : Found 33
> unprocessed instances
> 2019-08-06 04:04:20.373  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.507  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.518 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.559 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.608  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: model.Instance
> 2019-08-06 04:04:20.610  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.617  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.621 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.625 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.750  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: [..]
> 2019-08-06 04:04:20.751  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:30.017 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed
> instances
> 2019-08-06 04:04:30.020  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:40.020 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed
> instances
> 2019-08-06 04:04:40.027  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3] InstancesProcessor
>               : Body type is: model.Instance
>

It basically process 2 instances out of 33 available. And that's it. For the following interations, it merely logs a single Body type. Since it executes at least once, I reckon that "type-wise" I'm ok, else nothing would work. I suppose I'm running out of consumers because of some other route. But it still beats me because, as the logging shows, the instanceHandler bean does not hang.

Question: what can be wrong/faulty in my routing setup? I wanted to execute the instanceHandler in parallel, like X at a time. But one step at a time.

Any help or guidance or guess are mostly welcome. Even wild guesses.

Thanks in advance!

Luiz Eduardo Guida Valmont

(21) 2220-6223
(21) 98663-1337

ALIS - Soluções em Engenharia & Sistemas
Reply | Threaded
Open this post in threaded view
|

AW: Help Needed: Splitter and Parallel Processing

Burkard Stephan
In reply to this post by Luiz Eduardo Valmont
Hi

OK, here is my wild guess :-)


Based on your log output I suspect the following:

1. It starts and finds 33 instances to process
2. After processing the second instance, there is a problem
3. Could be that the log output "Handled one of: [..]" instead of "Handled one of: model.Instance" is an indicator for that.
4. Processing is aborted, perhaps the instanceHandler is swallowing exceptions
5. The next trigger starts, it still finds 33 instances because the already processed ones are not removed from the list of unprocessed
6. It tries to process the first, but fails because it is already processed.
7. Processing is aborted
8. Endless loop between #5 and #7

Cheers
Stephan



-----Ursprüngliche Nachricht-----
Von: Luiz Eduardo Valmont <[hidden email]>
Gesendet: Dienstag, 6. August 2019 06:25
An: [hidden email]
Betreff: Help Needed: Splitter and Parallel Processing

Hi, friends!

According to instructions found on this link (splitter) <https://camel.apache.org/splitter.html> and this other link (bean) <https://camel.apache.org/bean.html>, I wrote this (simple?) route:

  *public* *void* configure() *throws* Exception {

>     from( "quartz2://instancesProcessor?cron=" + quartz )
>       .routeId( "InstancesProcessor" )
>         .process( loadUnprocessedInstancesProcessor )
>         .split()
>           .body()
>           .log( LoggingLevel.WARN, "Body type is:
> ${body.getClass().getName()}" )
>           .bean( instanceHandler, "handle" )
>           .log( LoggingLevel.WARN, "Handled one of:
> ${body.getClass().getName()}" )
>         .end()
>       .end()
>     .end()
>     ;
>

And this is the general output:

2019-08-06 04:04:20.281 TRACE fcdc964e5ee8 --- [aultQuartzScheduler-Orthanc

> Overseer_Worker-1] .r.LoadUnprocessedInstancesProcessorImpl : Found 33
> unprocessed instances
> 2019-08-06 04:04:20.373  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.507  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.518 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.559 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.608  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: model.Instance
> 2019-08-06 04:04:20.610  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:20.617  INFO fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.h.TagInstanceHandler         : Tagged instance [...]
> 2019-08-06 04:04:20.621 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> b.c.a.r.o.o.s.InstanceServiceImpl        : Saving Instance [...]
> 2019-08-06 04:04:20.625 DEBUG fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1]
> c.a.r.o.o.h.SaveInstanceHandlerDecorator : Saved instance [...]
> 2019-08-06 04:04:20.750  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Handled one of: [..]
> 2019-08-06 04:04:20.751  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-1] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:30.017 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed
> instances
> 2019-08-06 04:04:30.020  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-2] InstancesProcessor
>               : Body type is: model.Instance
> 2019-08-06 04:04:40.020 TRACE fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3]
> .r.LoadUnprocessedInstancesProcessorImpl : Found 33 unprocessed
> instances
> 2019-08-06 04:04:40.027  WARN fcdc964e5ee8 ---
> [aultQuartzScheduler-Orthanc Overseer_Worker-3] InstancesProcessor
>               : Body type is: model.Instance
>

It basically process 2 instances out of 33 available. And that's it. For the following interations, it merely logs a single Body type. Since it executes at least once, I reckon that "type-wise" I'm ok, else nothing would work. I suppose I'm running out of consumers because of some other route. But it still beats me because, as the logging shows, the instanceHandler bean does not hang.

Question: what can be wrong/faulty in my routing setup? I wanted to execute the instanceHandler in parallel, like X at a time. But one step at a time.

Any help or guidance or guess are mostly welcome. Even wild guesses.

Thanks in advance!

Luiz Eduardo Guida Valmont

(21) 2220-6223
(21) 98663-1337

ALIS - Soluções em Engenharia & Sistemas