How to improve efficiency with Camel?

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

How to improve efficiency with Camel?

pmp.martins
This post was updated on .
I have a problem of efficiency in my project which uses Camel with the Esper component.

I have several external datasources feeding information to camel endpoints. Each Camel endpoint that receives data transfers it to a route that processes it and then delivers it at an Esper endpoint.

The image below illustrates this behavior:

Probleml

The efficiency problem is that all of this is done by a single Java thread. Thus if I have many sources, there is a huge bottleneck.


The following code accurately illustrates what is going on with the image:

   
public final void configure() throws OperationNotSupportedException{
        RouteDefinition route = from("xmpp://localhost:5222/?blablabla...");
        
        // apply some filter
        FilterDefinition filterDefinition = route.filter().method(...);
    		
        // apply main processor
        ExpressionNode expressionNode = filterDefinition.process(...);
    		
        
        // set destination
        expressionNode = filterDefinition.to("esper://session_X");
    }


To fix this problem, I have to handle this situation with a pool of threads or using some sort of parallel processing. I cannot use patterns like multicast, recipient list, etc because all of those send the same message to multiple endpoints / clients, which is not the case in my examples.

A possible solution would be having 1 thread per each "Datasource endpoint -> Route -> Esper endpoint" combination, like the image bellow:

solution_1

Another possible solution is to have 1 thread receive everything from the datasources, and then dispatch it to multiple threads handling the route processing together with the other endpoint:

solution_2

PS: I am open to any other possible suggestions you may have.

To achieve one of these I have considered using the Camel SEDA component component, however, this one does not seem to allow me to have dynamic thread pools, because the `concurrentConsumers` property is static.

Furthermore, I was alerted in StackOverflow for loss of messages using SEDA, so I am not sure how trustworthy this component really is. Can someone confirm?

So, at this point I am quite lost and I don't know what to do:
- Is SEDA the solution I am looking for?
- If yes, how do I integrate it with the Esper endpoint given the syntax problem?
- Are there any other solutions / Camel components that could fix my problem?
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

hekonsek
Hi,

> The efficiency problem is that all of this is done by a single Java thread.
> Thus if I have many sources, there is a huge bottleneck.

By default each consumer should generate incoming messages in the
separated thread. If you consume from two data source in two separated
routes, each of them should be processed by separated thread as well.
Can you show us your routes definitions?

Cheers.

--
Henryk Konsek
http://henryk-konsek.blogspot.com
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

pmp.martins
 You are correct, about your statement, I have just realized that the images are outdated. Since I am using Google Docs Drawing tool and since I am feeding this post with a direct URL I assumed that the images would be updated directly. Obviously I messed up and I apologize for the confusion.

The images are now fixed !
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

hekonsek
Hi,

> The images are now fixed !

Actually I still can't see the difference. :)

By default Camel processes each poll consumer thread (i.e. route) in
separated thread. If you define two DB consumers, they will poll the
database server concurrently. Please send us your route, so we could
tell you more about your case.

Cheers.

--
Henryk Konsek
http://henryk-konsek.blogspot.com
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

pmp.martins
The code of the route is provided in the configure() method, and my logs
show that 1 thread is handling both routes :S Do you want to see them?


2014-02-25 9:19 GMT+00:00 Henryk Konsek <[hidden email]>:

> Hi,
>
> > The images are now fixed !
>
> Actually I still can't see the difference. :)
>
> By default Camel processes each poll consumer thread (i.e. route) in
> separated thread. If you define two DB consumers, they will poll the
> database server concurrently. Please send us your route, so we could
> tell you more about your case.
>
> Cheers.
>
> --
> Henryk Konsek
> http://henryk-konsek.blogspot.com
>
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

hekonsek
Hi Pedro,

> The code of the route is provided in the configure() method,

But you got only single route configured there :) .

from(xmpp).filter(...).process(...).to("esper");

So the whole route will be executed in the single thread i.e. the same
thread will pick the message from XMPP client and send it to Esper.

If this is the problem, then you can break the processing of the route
into two asynchronous parts with the Threads [1] DSL, just as
demonstrated on the snippet below.

from(xmpp).threads(10).filter(...).process(...).to("esper");

If Threads DSL is used, XMPP won't wait until message has been
delivered to Esper, but instead will continue to consume the messages.

Cheers.

[1] http://camel.apache.org/async.html

--
Henryk Konsek
http://henryk-konsek.blogspot.com
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

pmp.martins
Oh nice ! Thanks for the help!
If I may ask, what is the difference between DSL and SEDA?
Reply | Threaded
Open this post in threaded view
|

Re: How to improve efficiency with Camel?

hekonsek
In reply to this post by hekonsek
> Damn, I tried your suggestion, but it doesn't work. This i what I have:

Try to "filter" messages before the "threads". Unfortunately Java
Threads DSL is a little bit less flexible than it's XML version.

> If I may ask, what is the difference between DSL and SEDA?

Threads DSL submits processing of the message to the dedicated task
executor and immediately starts to consume another message. That
basically means that from the "threads" point in the route messages
are processed concurrently.

SEDA decouples route processing into stages using blocking queues.
SEDA offers more options and is slightly more complicated than Threads
DSL.

If you can use threads DSL instead of SEDA, use it. This will keep
your route neat and tiny.

Cheers.

--
Henryk Konsek
http://henryk-konsek.blogspot.com