Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
I did write a simple app using Spring Boot (1.2.7.RELEASE) and Apache Camel (2.15.0). The app is simple and has only 1 route: a timer will invoke a method on a bean every 1s. The method invoked will use ProducerTemplate to ssh into a remote machine, execute a small script, and print out the output to the console. Simple, right?
However, when profiling this, I can see the number of threads! It seems like any threads created for the ssh are never killed, but parked instead. Because of that I run OOM pretty quickly with an error:

Exception in thread "Thread-341" java.lang.OutOfMemoryError: unable to create new native thread

More description and profiler results can be found on stackoverflow:
http://stackoverflow.com/questions/33671567/spring-boot-camel-producertemplate-memory-leak

I think this is a bug, as from the documentation I can't see that I am using the ProducerTemplate wrongly.
Or am I? If so, please tell me. If I am doing things right and it seems like a bug, I will submit it...
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

Jakub Korab
To work out if it's your use of the ProducerTemplate, try the following
route instead and see whether the leak persists:

from("timer://foo?period=1000")
     .transform().constant("/home/_username_/some_temp_script.sh")
.to("ssh://_remote.machine.url.here_?username=_username_&keyPairProvider=#keyPairProvider")
     .log("Received: ${body}");

On 12/11/15 14:35, codan84 wrote:

> I did write a simple app using Spring Boot (1.2.7.RELEASE) and Apache Camel
> (2.15.0). The app is simple and has only 1 route: a timer will invoke a
> method on a bean every 1s. The method invoked will use ProducerTemplate to
> ssh into a remote machine, execute a small script, and print out the output
> to the console. Simple, right?
> However, when profiling this, I can see the number of threads! It seems like
> any threads created for the ssh are never killed, but parked instead.
> Because of that I run OOM pretty quickly with an error:
>
> Exception in thread "Thread-341" java.lang.OutOfMemoryError: unable to
> create new native thread
>
> More description and profiler results can be found on stackoverflow:
> http://stackoverflow.com/questions/33671567/spring-boot-camel-producertemplate-memory-leak
>
> I think this is a bug, as from the documentation I can't see that I am using
> the ProducerTemplate wrongly.
> Or am I? If so, please tell me. If I am doing things right and it seems like
> a bug, I will submit it...
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
Hmm Interesting, this indeed fixes the problem. I am forced to use a producerTemplate tho... Any ideas what am I doing wrong?
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

Claus Ibsen-2
Maybe your bean with the producer template is not singleton scoped, so
spring creates a new instance per message.

On Thu, Nov 12, 2015 at 4:10 PM, codan84 <[hidden email]> wrote:
> Hmm Interesting, this indeed fixes the problem. I am forced to use a
> producerTemplate tho... Any ideas what am I doing wrong?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773747.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
In that case starting/stopping it should in theory release threads, right?
I tried start->requestBody->stop, without effect, still all threads keep running. I also tried to instantiate new template for each call, start it and stop at the end, still the same.
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
In reply to this post by Claus Ibsen-2
I have tried similar thing, but with sftp rather than ssh. Everything else is exactly the same:

producerTemplate.sendBodyAndHeader(
                "sftp://_target_machine_url_?username=_username_" +
                        "&privateKeyFile=" + sshKeyPath +
                        "&knownHostsFile=" + knownHosts +
                        "&preferredAuthentications=publickey",
                new ByteArrayInputStream(new String("Random junk").getBytes()),
                "CamelFileName", "/home/_username_/test_file.txt"
        );

And this works fine. So it is not the way I am using ProducerTemplate. It is a bug with ssh when using producer template, or "requestBody" method...
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

Jakub Korab
In reply to this post by codan84
When you use the SSH producer endpoint in a to(..) statement, one
instance of the endpoint is created within the route, its lifecycle is
tied to the CamelContext and any resources used by it will be cleaned up
at shutdown. The endpoint creates an instance of an SshClient
(underlying library), and closes it when the context shuts down.

When you use it through a ProducerTemplate, each time you access the
send() method it creates a new instance of the SshEndpoint, allocating
memory on the way by creating a new underlying SshClient instance. This
behaviour is dictated by SshEndpoint. isSingleton(), which returns false
as the underlying client library is not thread-safe. You need to close
the ProducerTemplate in order to close the SshClient instance (note,
nothing to do with the SSH connection itself).

As a workaround, you might consider having the CamelContext injected
into your bean, and for each request do:

ProducerTemplate template = context.createProducerTemplate();
template.sendBody(...);
template.close();

It's expensive, but it will do the job.

I don't think this is a bug, if seems to be a side-effect of how a
ProducerTemplate works with non-singleton endpoints. If there were
multiple threads, then the ProducerTemplate would have to create
multiple endpoints for the underlying library to work correctly, and for
the threads to not trip over each other. In your case, only one thread
will ever be using that one ProducerTemplate, so it doesn't make sense.
The ProducerTemplate can't know in advance how many threads will be
accessing it.

On 12/11/15 15:10, codan84 wrote:
> Hmm Interesting, this indeed fixes the problem. I am forced to use a
> producerTemplate tho... Any ideas what am I doing wrong?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773747.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
I tried your suggestion:

        ProducerTemplate producerTemplate = camelContext.createProducerTemplate();
        producerTemplate.start();
        String response = producerTemplate.requestBody("ssh://....", String.class);
        producerTemplate.stop();

There is no difference. Still the number of threads grows.
I would disagree that this is not a bug. Especially with a Spring-Boot application, where the app will usually run on a server forever, this is an issue. It is unfortunate how Camel handles non-singleton endpoints, but it should either be very well documented (bold, red etc), or a different approach to non-singleton endpoints worked out.
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

Jakub Korab
If the threads are not being cleaned up, then it would seem that there
is a leak in the SshClient library - this is confirmed by the screenshot
of the threads that you attached on SO; they are all associated with
SshClient. Can you verify that the client is in fact shutting down by
set the logging level for org.apache.sshd.common.util to DEBUG?

As to the documentation around the ProducerTemplate and singletons,
absolutely. As they say, contributions are most welcome.

On 12/11/15 16:15, codan84 wrote:

> I tried your suggestion:
>
>          ProducerTemplate producerTemplate =
> camelContext.createProducerTemplate();
>          producerTemplate.start();
>          String response = producerTemplate.requestBody("ssh://....",
> String.class);
>          producerTemplate.stop();
>
> There is no difference. Still the number of threads grows.
> I would disagree that this is not a bug. Especially with a Spring-Boot
> application, where the app will usually run on a server forever, this is an
> issue. It is unfortunate how Camel handles non-singleton endpoints, but it
> should either be very well documented (bold, red etc), or a different
> approach to non-singleton endpoints worked out.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773755.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

codan84
Indeed the ClientSession of SshClient is being started, but never finished. Here are logs:

2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-1] INFO  o.a.s.c.session.ClientSessionImpl - Client session created
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-2] INFO  o.a.s.c.session.ClientSessionImpl - Server version string: SSH-1.99-OpenSSH_3.9p1
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO  o.a.s.c.session.ClientSessionImpl - Kex: server->client aes128-ctr hmac-sha1 none
2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO  o.a.s.c.session.ClientSessionImpl - Kex: client->server aes128-ctr hmac-sha1 none
2015-11-12 16:39:09 [sshd-SshClient[68c05218]-nio2-thread-5] WARN  o.a.s.c.k.AcceptAllServerKeyVerifier - Server at *** presented unverified DSA key: ***
----
Script output :)
----

This just repeats for every call.
Reply | Threaded
Open this post in threaded view
|

Re: Spring-Boot + Camel + producerTemplate ssh spawning thousands of threads

Jakub Korab
This bit is definitely a bug. Whether it's something in the component,
or the SshClient library needs to be investigated. You should log this
one in a Jira.

On 12/11/15 16:42, codan84 wrote:

> Indeed the ClientSession of SshClient is being started, but never finished.
> Here are logs:
>
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-1] INFO
> o.a.s.c.session.ClientSessionImpl - Client session created
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-2] INFO
> o.a.s.c.session.ClientSessionImpl - Server version string:
> SSH-1.99-OpenSSH_3.9p1
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
> o.a.s.c.session.ClientSessionImpl - Kex: server->client aes128-ctr hmac-sha1
> none
> 2015-11-12 16:39:08 [sshd-SshClient[68c05218]-nio2-thread-3] INFO
> o.a.s.c.session.ClientSessionImpl - Kex: client->server aes128-ctr hmac-sha1
> none
> 2015-11-12 16:39:09 [sshd-SshClient[68c05218]-nio2-thread-5] WARN
> o.a.s.c.k.AcceptAllServerKeyVerifier - Server at *** presented unverified
> DSA key: ***
> ----
> Script output :)
> ----
>
> This just repeats for every call.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Spring-Boot-Camel-producerTemplate-ssh-spawning-thousands-of-threads-tp5773741p5773762.html
> Sent from the Camel - Users mailing list archive at Nabble.com.