Camel's use of jsch/java.io

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Camel's use of jsch/java.io

neil
Hi,
I would like to report a potential bug in Camel's use of jsch/java.io, and find a potential workaround.  

Apologies in advance for abusing terms -- I am only a Camel user, I am not intimately familiar with its implementation.

We have an indefinite polling route that does a directory listing on a third-party's sftp server, and subsequently downloads "new" files and does things with them.

A couple times per year, this route halts apparent operation with no error logged, and no impact to other jvm activities.  The impact of this behavior is dire to us -- there is no error reported, yet Camel is not doing anything.

Looking at a thread dump, the "sftp" camel thread is stuck in an apparent infinite loop in java.io.PipedInputStream.read().  Ignoring the deeply weird use of thread safety in this class for a moment, it appears that this can happen if Camel moves its processor instance of this sftp route to a different thread while it is processing.

public synchronized int read() throws IOException {
    if (!connected) {
        throw new IOException("Pipe not connected");
    } else if (closedByReader) {
        throw new IOException("Pipe closed");
    } else if (writeSide != null && !writeSide.isAlive()
        && !closedByWriter && (in < 0)) {
        throw new IOException("Write end dead");
    }

    readSide = Thread.currentThread();
    int trials = 2;
    while (in < 0) {
        if (closedByWriter) {
            /* closed by writer, return EOF */
            return -1;
        }
        if ((writeSide != null) && (!writeSide.isAlive()) && (--trials < 0)) {  // <----- If writeSide is null, infinite loop?
            throw new IOException("Pipe broken");
        }
        /* might be a writer waiting */
        notifyAll();
        try {
            wait(100);  // <----  MY THREAD SPENDS 100% TIME HERE
        } catch (InterruptedException ex) {
            throw new java.io.InterruptedIOException();
        }
    }
    int ret = buffer[out++] & 0xFF;
    if (out >= buffer.length) {
        out = 0;
    }
    if (in == out) {
        /* now empty */
        in = -1;
    }
    return ret;
}


Might that be the case?  If it is, what is the recommended way of tearing down a route and then rebuilding it?  Most of the "shutdown" docs are aimed at shutting down an entire container, which we most certainly do not want to do.

And should camel do this by default on routes that use jsch/java.io.PipedInputStream as a transport implementation?  It seems like the resources required to spin up a new thread would be dwarfed by creating a new ssh connection.

If that is not the case, is there any other known issue about polling sftp routes?  I've googled around and I have not found anything, yet we have seen this half a dozen times over the last few months.

Any advice is appreciated!

Thanks!
-neil
Reply | Threaded
Open this post in threaded view
|

Re: Camel's use of jsch/java.io

Claus Ibsen-2
Hi

The code is in jsch? If so you should report this to the jsch team.

On Tue, Oct 1, 2013 at 6:46 AM, neil <[hidden email]> wrote:

> Hi,
> I would like to report a potential bug in Camel's use of jsch/java.io, and
> find a potential workaround.
>
> Apologies in advance for abusing terms -- I am only a Camel user, I am not
> intimately familiar with its implementation.
>
> We have an indefinite polling route that does a directory listing on a
> third-party's sftp server, and subsequently downloads "new" files and does
> things with them.
>
> A couple times per year, this route halts apparent operation with no error
> logged, and no impact to other jvm activities.  The impact of this behavior
> is dire to us -- there is no error reported, yet Camel is not doing
> anything.
>
> Looking at a thread dump, the "sftp" camel thread is stuck in an apparent
> infinite loop in java.io.PipedInputStream.read().  Ignoring the deeply weird
> use of thread safety in this class for a moment, it appears that this can
> happen if Camel moves its processor instance of this sftp route to a
> different thread while it is processing.
>
> public synchronized int read() throws IOException {
>     if (!connected) {
>         throw new IOException("Pipe not connected");
>     } else if (closedByReader) {
>         throw new IOException("Pipe closed");
>     } else if (writeSide != null && !writeSide.isAlive()
>         && !closedByWriter && (in < 0)) {
>         throw new IOException("Write end dead");
>     }
>
>     readSide = Thread.currentThread();
>     int trials = 2;
>     while (in < 0) {
>         if (closedByWriter) {
>             /* closed by writer, return EOF */
>             return -1;
>         }
>         if ((writeSide != null) && (!writeSide.isAlive()) && (--trials < 0))
> {  // <----- If writeSide is null, infinite loop?
>             throw new IOException("Pipe broken");
>         }
>         /* might be a writer waiting */
>         notifyAll();
>         try {
>             wait(100);  // <----  MY THREAD SPENDS 100% TIME HERE
>         } catch (InterruptedException ex) {
>             throw new java.io.InterruptedIOException();
>         }
>     }
>     int ret = buffer[out++] & 0xFF;
>     if (out >= buffer.length) {
>         out = 0;
>     }
>     if (in == out) {
>         /* now empty */
>         in = -1;
>     }
>     return ret;
> }
>
>
> Might that be the case?  If it is, what is the recommended way of tearing
> down a route and then rebuilding it?  Most of the "shutdown" docs are aimed
> at shutting down an entire container, which we most certainly do not want to
> do.
>
> And should camel do this by default on routes that use
> jsch/java.io.PipedInputStream as a transport implementation?  It seems like
> the resources required to spin up a new thread would be dwarfed by creating
> a new ssh connection.
>
> If that is not the case, is there any other known issue about polling sftp
> routes?  I've googled around and I have not found anything, yet we have seen
> this half a dozen times over the last few months.
>
> Any advice is appreciated!
>
> Thanks!
> -neil
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-s-use-of-jsch-java-io-tp5740623.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: Camel's use of jsch/java.io

neil
The code that pointed out is in java.io.PipedInputStream.read().

I believe Camel's thread management is triggering an infinite loop in it, although I am not certain.

Is there a graceful way to have a Camel polling route completely reset itself periodically?  Or is bouncing the container a better option?

Thanks,
-neil
Reply | Threaded
Open this post in threaded view
|

Re: Camel's use of jsch/java.io

Bengt Rodehav
Hello Neil,

I use the sftp component quite a lot. Now and then (perhaps once in a
month) it stops polling. I think the problem to that lies in the Jsch
itself and not in Camel. I'm not sure if it is the same problem you have
though.

I also took stack traces and found that Jsch was stuck in a read operation
but I think that was a socket read operation. It seems like Jsch does not
use TCP keepalive. This means that if the other end disconnects (or someone
in between), Jsch will never be notified and hangs forever in a read.

Instead Jsch relies on a timeout higher up in the protocol level (it might
be part of SSH I'm not sure). This timeout can be set by specifying the
"serverAliveInterval" parameter in the sftp component.

Using a polling delay of 60000 ms, we now set the serverAliveInterval to
10000ms. We haven't had any problems since (no proof though since we've
only been doing it for less than three months).

Although it isn't necessarily the same problem, it might still be worth
trying the serverAliveInterval parameter.

/Bengt


2013/10/1 neil <[hidden email]>

> The code that pointed out is in java.io.PipedInputStream.read().
>
> I believe Camel's thread management is triggering an infinite loop in it,
> although I am not certain.
>
> Is there a graceful way to have a Camel polling route completely reset
> itself periodically?  Or is bouncing the container a better option?
>
> Thanks,
> -neil
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Camel-s-use-of-jsch-java-io-tp5740623p5740700.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>