Exception FTP Component

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Exception FTP Component

Niels
Hey everybody,

I'm using to ftp component to consume from a ftp server.
The little application seems to do the job but when investigating the log files this exception shows up:

Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
 Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
 FTP response 421 received.  Server closed connection.. Code: 421. This exception will be ignored.
 Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
 FTP response 421 received.  Server closed connection.. Code: 421
 org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
 FTP response 421 received.  Server closed connection.. Code: 421
        at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
        at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received.  Server closed connection.
        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
        at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
        at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
                at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
                ... 13 more

This is the route I'm using to consume the service:

from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
                        .filter(xmlPredicate)
                                .to("jms:xmlQueue").end()
                        .filter(imgPredicate)
                                .to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
                        .filter(pdfPredicate)
                                .to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
                        .to("file://" + props.getProperty(ROOT_DIR) + "/junk");

Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
Therefore I'm asking some experts opinion on my issue.

And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.

Thanks!

Regards,

Niels


Reply | Threaded
Open this post in threaded view
|

Re: Exception FTP Component

Claus Ibsen-2
Hi

Just a note about your Java DSL route.

What you do looks much more like the Content Based Router (eg if ...
else if ... else if ... else).
http://camel.apache.org/content-based-router

So you may want to look into that instead of using the filter EIP if
you think the code is more readable.


And in terms of the FTP error. What version of Camel are you using? We
have fixed some pieces in recent times about detecting
broken connection and re-connection.

If you do not want to keep the connection alive, you can use the
disconnect=true option. Then you force Camel to re-login on each poll.
This could make sense if there is a long delay between polls.



On Mon, Jan 16, 2012 at 9:25 AM, Niels <[hidden email]> wrote:

> Hey everybody,
>
> I'm using to ftp component to consume from a ftp server.
> The little application seems to do the job but when investigating the log files this exception shows up:
>
> Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
>  Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
>  FTP response 421 received.  Server closed connection.. Code: 421. This exception will be ignored.
>  Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
>  FTP response 421 received.  Server closed connection.. Code: 421
>  org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
>  FTP response 421 received.  Server closed connection.. Code: 421
>        at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
>        at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
>        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
>        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
>        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>        at java.lang.Thread.run(Thread.java:662)
> Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received.  Server closed connection.
>        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
>        at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
>        at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
>                at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
>                ... 13 more
>
> This is the route I'm using to consume the service:
>
> from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
>                        .filter(xmlPredicate)
>                                .to("jms:xmlQueue").end()
>                        .filter(imgPredicate)
>                                .to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
>                        .filter(pdfPredicate)
>                                .to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
>                        .to("file://" + props.getProperty(ROOT_DIR) + "/junk");
>
> Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
> Therefore I'm asking some experts opinion on my issue.
>
> And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.
>
> Thanks!
>
> Regards,
>
> Niels
>
>



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|

Re: Exception FTP Component

Niels
Thanks for the fast reply.

I already changed to content-based routing.
I'm using the latest version of camel 2.9.0.
It's better to use the disconnect=true option like you mentioned, because there is a 5 min interval between the polling.
Will update a soon as I can redeploy my application en re-encounter the same problem in the logs.

Regards,

Niels

On Jan 16, 2012, at 11:46 AM, Claus Ibsen wrote:

> Hi
>
> Just a note about your Java DSL route.
>
> What you do looks much more like the Content Based Router (eg if ...
> else if ... else if ... else).
> http://camel.apache.org/content-based-router
>
> So you may want to look into that instead of using the filter EIP if
> you think the code is more readable.
>
>
> And in terms of the FTP error. What version of Camel are you using? We
> have fixed some pieces in recent times about detecting
> broken connection and re-connection.
>
> If you do not want to keep the connection alive, you can use the
> disconnect=true option. Then you force Camel to re-login on each poll.
> This could make sense if there is a long delay between polls.
>
>
>
> On Mon, Jan 16, 2012 at 9:25 AM, Niels <[hidden email]> wrote:
>> Hey everybody,
>>
>> I'm using to ftp component to consume from a ftp server.
>> The little application seems to do the job but when investigating the log files this exception shows up:
>>
>> Trying to recover by disconnecting from remote server forcing a re-connect at next poll: ftp://userl@server:21
>>  Error occurred while disconnecting from ftp://user@server:21 due: File operation failed: 421 Terminating connection.
>>  FTP response 421 received.  Server closed connection.. Code: 421. This exception will be ignored.
>>  Consumer FtpConsumer[ftp://server?delay=900000&delete=true&password=******&username=user] could not poll endpoint: Endpoint[ftp://server?de$
>>  FTP response 421 received.  Server closed connection.. Code: 421
>>  org.apache.camel.component.file.GenericFileOperationFailedException: File operation failed: 421 Timeout (120 seconds): closing control connection.
>>  FTP response 421 received.  Server closed connection.. Code: 421
>>        at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:569)
>>        at org.apache.camel.component.file.remote.FtpConsumer.pollDirectory(FtpConsumer.java:45)
>>        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:100)
>>        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
>>        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
>>        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
>>        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
>>        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
>>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
>>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
>>        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>>        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>>        at java.lang.Thread.run(Thread.java:662)
>> Caused by: org.apache.commons.net.ftp.FTPConnectionClosedException: FTP response 421 received.  Server closed connection.
>>        at org.apache.commons.net.ftp.FTP.__getReply(FTP.java:347)
>>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:495)
>>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:537)
>>        at org.apache.commons.net.ftp.FTP.sendCommand(FTP.java:586)
>>        at org.apache.commons.net.ftp.FTP.pwd(FTP.java:1381)
>>        at org.apache.commons.net.ftp.FTPClient.printWorkingDirectory(FTPClient.java:1990)
>>                at org.apache.camel.component.file.remote.FtpOperations.getCurrentDirectory(FtpOperations.java:567)
>>                ... 13 more
>>
>> This is the route I'm using to consume the service:
>>
>> from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&delete=true&delay=" + props.getProperty(FTP_DELAY))
>>                        .filter(xmlPredicate)
>>                                .to("jms:xmlQueue").end()
>>                        .filter(imgPredicate)
>>                                .to("file://" + props.getProperty(ROOT_DIR) + "/img").end()
>>                        .filter(pdfPredicate)
>>                                .to("file://" + props.getProperty(ROOT_DIR) + "/pdf").end()
>>                        .to("file://" + props.getProperty(ROOT_DIR) + "/junk");
>>
>> Now I'm wondering if this could be an issue with camel? In my opinion I think it has something to do with the FTP server itself and not so much with camel.
>> Therefore I'm asking some experts opinion on my issue.
>>
>> And please if you have any remarks on my code, I happy to here them I'm just a novice when it comes to camel.
>>
>> Thanks!
>>
>> Regards,
>>
>> Niels
>>
>>
>
>
>
> --
> Claus Ibsen
> -----------------
> FuseSource
> Email: [hidden email]
> Web: http://fusesource.com
> Twitter: davsclaus, fusenews
> Blog: http://davsclaus.blogspot.com/
> Author of Camel in Action: http://www.manning.com/ibsen/

Reply | Threaded
Open this post in threaded view
|

Re: Exception FTP Component

Niels
I changed my routing and added a catch for an exception.

I thought everything was running fine but apparently it doesn't.
I still get this exception:

Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: GenericFile[DZ_20120104_DZ_004.PDF] from: Endpoint[ftp://server?delay=300000&delete=true&disconnect=true&password=******&username=user]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: GenericFile[DZ_20120104_DZ_004.PDF] from: Endpoint[ftp://server?delay=300000&delete=true&disconnect=true&password=******&username=user]
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:338)
        at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:94)
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:175)
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:136)
        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:140)
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

I changed my route to this:
         //XML Predicate
        //only allows names without spaces
                Predicate xmlPredicate = header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(xml))$)");
                //Images Predicate
        //only allows names without spaces
                Predicate imgPredicate = header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(jpg|png|gif))$)");
                //PDF Predicate
        //only allows names without spaces
                Predicate pdfPredicate = header(CAMEL_FILE_NAME).regex("([\\S]+(\\.(?i)(pdf))$)");

                onException(SchemaValidationException.class)
                        .to("file://" + props.getProperty(ROOT_DIR) + "/invalid");
               
                from("ftp://" + props.getProperty(FTP_URL) + "?username=" + props.getProperty(FTP_USER) + "&password=" + props.getProperty(FTP_PWD) + "&disconnect=true&delete=true&delay=" + props.getProperty(FTP_DELAY))
                        .choice()
                                .when(xmlPredicate)
                                        .to("jms:xmlQueue")
                                .when(imgPredicate)
                                        .to("file://" + props.getProperty(ROOT_DIR) + "/img")
                                .when(pdfPredicate)
                                        .to("file://" + props.getProperty(ROOT_DIR) + "/pdf")
                                .otherwise()
                                        .to("file://" + props.getProperty(ROOT_DIR) + "/junk");
               
                from("jms:xmlQueue")
                                    .to("validator:FtpXmlValidator.xsd")
                    .to("xslt://XmlToRssConverter.xsl")
                    .to("file://" + props.getProperty(ROOT_DIR) + "/rss");

It was already running for a week now. This error occured when I dropped like 42 pdf's (size between 5- 10 mb) on the ftp server.

But the days before it processed 435 files, sporadicly added to the server, like 5 at a time or less.
It seems like the ftp has problems when it has to fetch a large numbers of files. When I first run camel, it started to process the ftp server, at the moment there were 1200 files stored on it. All xml files or small images, so is it running out of memory or what is causing this exception.

Do I need to add more memory to the JVM? Or is it something else?

Like I mentioned before I'm using :
Debian 5.0

java version "1.6.0_26"
Java(TM) SE Runtime Environment (build 1.6.0_26-b03)
Java HotSpot(TM) Client VM (build 20.1-b02, mixed mode, sharing)

Hopefully you guys have some advise

regards

Camel: 2.9.0