FTP performance with hundreds of concurrent routes.

[domenec]

Hello and thanks beforehand for any response, pointer or simply reading.

Without previous knowledge of Camel I was requested to support a customer who had "a Java application that does FTP's at scheduled times and eats up server resources, and they have no source code".

OK, I inspected the WAR, found out it was Camel, discovered what it was (and liked it, I have done MQ Series and Oracle Service Bus and several integration projects).

At the moment the Tomcat server has been instrumented with JMX and I will collect data about threads, memory usage, whatever gives some hint.

I have a suspect on threading, based a priori on FTP usually eating two threads (for data and communication channels) and a posteriori for having read about performance issues with default pool of 10 threads, which were fixed by going to 50 threads.

So, at the moment all I can say is that there are about 600 to 800 possibly concurrent FTP's, which looks like a lot for 10 threads, if I get it right.

No questions, I'd just appreciate hints on what to look and report if it looks suspicious, based on your experience, thanks :)

[Claus Ibsen-2]

On Wed, May 9, 2012 at 2:29 PM, domenec <[hidden email]> wrote:

> Hello and thanks beforehand for any response, pointer or simply reading.
>
> Without previous knowledge of Camel I was requested to support a customer
> who had "a Java application that does FTP's at scheduled times and eats up
> server resources, and they have no source code".
>
> OK, I inspected the WAR, found out it was Camel, discovered what it was (and
> liked it, I have done MQ Series and Oracle Service Bus and several
> integration projects).
>
> At the moment the Tomcat server has been instrumented with JMX and I will
> collect data about threads, memory usage, whatever gives some hint.
>
> I have a suspect on threading, based a priori on FTP usually eating two
> threads (for data and communication channels) and a posteriori for having
> read about performance issues with default pool of 10 threads, which were
> fixed by going to 50 threads.
>
> So, at the moment all I can say is that there are about 600 to 800 possibly
> concurrent FTP's, which looks like a lot for 10 threads, if I get it right.
>
> No questions, I'd just appreciate hints on what to look and report if it
> looks suspicious, based on your experience, thanks :)
>

What version of Camel do you use?

And do you download or upload files to the FTP server? (consumer or
producer in EIP terms)

The Camel FTP consumer is by default single threaded, eg

So if you have a route that does: (pseudo)
from ftp
  to bean

Then only 1 thread is active.



>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/

[Rocco Gallo Citera]

If there were a threading issue with Camel, you should see clearly a
bottleneck in the FTP, since you need to move multiple files with just
10 threads. What kind of resources have a problem? Can you isolate the
war in another enviroment and replicate the problem?

On Wed, May 9, 2012 at 9:47 AM, Claus Ibsen <[hidden email]> wrote:

> On Wed, May 9, 2012 at 2:29 PM, domenec <[hidden email]> wrote:
>> Hello and thanks beforehand for any response, pointer or simply reading.
>>
>> Without previous knowledge of Camel I was requested to support a customer
>> who had "a Java application that does FTP's at scheduled times and eats up
>> server resources, and they have no source code".
>>
>> OK, I inspected the WAR, found out it was Camel, discovered what it was (and
>> liked it, I have done MQ Series and Oracle Service Bus and several
>> integration projects).
>>
>> At the moment the Tomcat server has been instrumented with JMX and I will
>> collect data about threads, memory usage, whatever gives some hint.
>>
>> I have a suspect on threading, based a priori on FTP usually eating two
>> threads (for data and communication channels) and a posteriori for having
>> read about performance issues with default pool of 10 threads, which were
>> fixed by going to 50 threads.
>>
>> So, at the moment all I can say is that there are about 600 to 800 possibly
>> concurrent FTP's, which looks like a lot for 10 threads, if I get it right.
>>
>> No questions, I'd just appreciate hints on what to look and report if it
>> looks suspicious, based on your experience, thanks :)
>>
>
> What version of Camel do you use?
>
> And do you download or upload files to the FTP server? (consumer or
> producer in EIP terms)
>
> The Camel FTP consumer is by default single threaded, eg
>
> So if you have a route that does: (pseudo)
> from ftp
>  to bean
>
> Then only 1 thread is active.
>
>
>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
> FuseSource
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/

[domenec]

Claus Ibsen-2 wrote

What version of Camel do you use?
And do you download or upload files to the FTP server? (consumer or
producer in EIP terms)
The Camel FTP consumer is by default single threaded, eg
So if you have a route that does: (pseudo)
from ftp
  to bean
Then only 1 thread is active.

Thanks for the response (you and Rocco). There have been holidays in between, please excuse the time gone so far.

Camel version in WEB-INF/lib is *-2.7.2.jar (although they left an strange WEB-INF/lib/camel/*-2.7.1.jar subdir).

Concerning routes there are:

- 140 with from uri="ftp: to a file
- Another 300 of a local file to an ftp uri

I have no idea of what average file size is. Someone else developed, I was asked to take a look.

Is it possible that being single threaded and having many FTP transfers waiting in parallel makes the CPU go that high? See the bursts in the slides I made next.

I did a JMX capture of one night of activity which includes some coments and thread count over time and their state:
http://www.slideshare.net/domenex/jmx-capture

Thanks for any hint. Beer tokens in Barcelona offered in exchange :)

[Claus Ibsen-2]

On Mon, Jul 9, 2012 at 6:31 PM, domenec <[hidden email]> wrote:

>
> Claus Ibsen-2 wrote
>>
>> What version of Camel do you use?
>> And do you download or upload files to the FTP server? (consumer or
>> producer in EIP terms)
>> The Camel FTP consumer is by default single threaded, eg
>> So if you have a route that does: (pseudo)
>> from ftp
>>   to bean
>> Then only 1 thread is active.
>>
>
> Thanks for the response (you and Rocco). There have been holidays in
> between, please excuse the time gone so far.
>
> Camel version in WEB-INF/lib is *-2.7.2.jar (although they left an strange
> WEB-INF/lib/camel/*-2.7.1.jar subdir).
>
> Concerning routes there are:
>
> - 140 with from uri="ftp: to a file
> - Another 300 of a local file to an ftp uri
>
> I have no idea of what average file size is. Someone else developed, I was
> asked to take a look.
>
> Is it possible that being single threaded and having many FTP transfers
> waiting in parallel makes the CPU go that high? See the bursts in the slides
> I made next.
>

Yeah each from has its own thread, so you have 140 + 300 = 440 threads.

Although they dont run at the same time. You can control how often
they run, and by default they run 2 times per second, which is very
fast. So for production usage you should raise this delay, by setting
delay to a higher value . You can have different values, so some
routes can run more frequently if they are more important to pickup
new files asap.

Also you should set the option useFixedDelay=true, when you have so
many routes. You can read about this option here, and the JDK javadoc
what it means.
http://camel.apache.org/file2

And from Camel 2.10 onwards you can configure a shared thread pool for
all your file/ftp consumers. So instead of having 440 threads, you can
have a thread pool with a max limit. Also with a pool the threads can
be shared among the consumers etc. Its the new
scheduledExecutorService option

> I did a JMX capture of one night of activity which includes some coments and
> thread count over time and their state:
> http://www.slideshare.net/domenex/jmx-capture
>
> Thanks for any hint. Beer tokens in Barcelona offered in exchange :)
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/FTP-performance-with-hundreds-of-concurrent-routes-tp5697247p5715757.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen

[domenec]

Thanks for the tips, Claus.