Quantcast

Slow startup of routes

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Slow startup of routes

hussainpirosha
Hi,

I am using camel to implement dispatcher EIP. There are thousands of messages in a queue which needs to be delivered at different URLs. Each message has its own delivery URL and delivery protocol (ftp,email,http etc).

The way we are implementing in Camel is
- Boot a single camel context, the context is disabled for JMX and the loadStatisticsEnabled is set to false on the ManagementStrategy. As mentioned in a jira issue, addressed in 2.11.0 version, for disabling the background management thread creation.
- For each message a route is being constructed , the message is being pushed to the route for delivery.
- After the message is processed route is shutdown and removed from context.

We did a small perf test by having 200 threads of dispatcher component, each sharing the same context.
Observed that the time to start a route increases upto a maximum of 60 seconds while the time to process is in milliseconds.

Issue CAMEL-5675 mentions that this has been fixed but we are still observing significant time being taken in starting up routes.

Pls suggest, what should we do to solve this.

The route that we are creating for http is

from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true", dispatchItem.getEndPointURL(),timeOutInMillis);

Each dispatchItem has a unique ID.

Thanks,
Hussain


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Slow startup of routes

Willem.Jiang
Hi,

It looks like Camel takes lots of time to create a new Http endpoint, maybe you need to consider to do some caching work for it.
BTW, whyou you need to se the soTimeout per endpoint.


--  
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem





On Sunday, May 12, 2013 at 3:10 PM, hussainpirosha wrote:

> Hi,
>  
> I am using camel to implement dispatcher EIP. There are thousands of
> messages in a queue which needs to be delivered at different URLs. Each
> message has its own delivery URL and delivery protocol (ftp,email,http etc).
>  
> The way we are implementing in Camel is  
> - Boot a single camel context, the context is disabled for JMX and the
> loadStatisticsEnabled is set to false on the ManagementStrategy. As
> mentioned in a jira issue, addressed in 2.11.0 version, for disabling the
> background management thread creation.
> - For each message a route is being constructed , the message is being
> pushed to the route for delivery.
> - After the message is processed route is shutdown and removed from context.
>  
> We did a small perf test by having 200 threads of dispatcher component, each
> sharing the same context.
> Observed that the time to start a route increases upto a maximum of 60
> seconds while the time to process is in milliseconds.  
>  
> Issue CAMEL-5675 mentions that this has been fixed but we are still
> observing significant time being taken in starting up routes.
>  
> Pls suggest, what should we do to solve this.
>  
> The route that we are creating for http is  
>  
> from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true",
> dispatchItem.getEndPointURL(),timeOutInMillis);
>  
> Each dispatchItem has a unique ID.
>  
> Thanks,
> Hussain
>  
>  
>  
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).



Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Slow startup of routes

hussainpirosha
Thanks Willem,

I am not clear how can we use caching for component creation, please elaborate on it.

Need to set the soTimeout per endpoint since each dispatch message contain its endpoint and timeOut settings.

Thanks,
Hussain
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Slow startup of routes

Claus Ibsen-2
In reply to this post by hussainpirosha
Hi

Your design is probably not good when you need to create a new route
to process each message.

Instead reuse the route(s) and use the dynamic EIPs if you need to
process the messages a bit differently.
http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html


On Sun, May 12, 2013 at 9:10 AM, hussainpirosha
<[hidden email]> wrote:

> Hi,
>
> I am using camel to implement dispatcher EIP. There are thousands of
> messages in a queue which needs to be delivered at different URLs. Each
> message has its own delivery URL and delivery protocol (ftp,email,http etc).
>
> The way we are implementing in Camel is
> - Boot a single camel context, the context is disabled for JMX and the
> loadStatisticsEnabled is set to false on the ManagementStrategy. As
> mentioned in a jira issue, addressed in 2.11.0 version, for disabling the
> background management thread creation.
> - For each message a route is being constructed , the message is being
> pushed to the route for delivery.
> - After the message is processed route is shutdown and removed from context.
>
> We did a small perf test by having 200 threads of dispatcher component, each
> sharing the same context.
> Observed that the time to start a route increases upto a maximum of 60
> seconds while the time to process is in milliseconds.
>
> Issue CAMEL-5675 mentions that this has been fixed but we are still
> observing significant time being taken in starting up routes.
>
> Pls suggest, what should we do to solve this.
>
> The route that we are creating for http is
>
> from("direct:"+dispatchItem.getID()).toF("%s?httpClient.soTimeout=%s&disableStreamCache=true",
> dispatchItem.getEndPointURL(),timeOutInMillis);
>
> Each dispatchItem has a unique ID.
>
> Thanks,
> Hussain
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Slow-startup-of-routes-tp5732356.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Slow startup of routes

hussainpirosha
Thanks Claus for pointing out the mistake.

Changed the design to have a single route per protocol and now using the dynamic EIP pattern to pass in the endpoint url and other parameters. The new route looks like :

from("direct:http")
        .recipientList(simple("http:${header.url}? httpClient.soTimeout=${header.timeout}&disableStreamCache=true"));


Thanks,
Hussain




Loading...