out of heap space if remote FTP site has too many files to pick up

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

out of heap space if remote FTP site has too many files to pick up

skasish
out of heap space if remote FTP site has too many files to pick up.

Is there configuration problem.

My simple code is given below

import org.slf4j.*;
import org.apache.camel.*;
import org.apache.camel.builder.*;
import org.apache.camel.main.Main;
import org.apache.camel.component.dataset.*;

public class FileDemoCamel extends Main {
    static Logger LOG = LoggerFactory.getLogger(FileDemoCamel.class);
    public static void main(String[] args) throws Exception {
        FileDemoCamel main = new FileDemoCamel();
        main.enableHangupSupport();
        main.addRouteBuilder(createRouteBuilder());
        main.bind("sampleGenerator", createDataSet());
        main.run(args);
    }
    static RouteBuilder createRouteBuilder() {
        return new RouteBuilder() {
            public void configure() {
                from("ftp://admin@localhost:2121/MF?noop=true&password=admin&passiveMode=true&maxMessagesPerPoll=10000&doneFileName=${file:name.noext}.TRG")
                .to("file://D:/temp");
            }
        };
    }
    static DataSet createDataSet() {
        return new SimpleDataSet();
    }
}
Reply | Threaded
Open this post in threaded view
|

Re: out of heap space if remote FTP site has too many files to pick up

skasish
Following error has been occurred

13:39:40.609 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] DEBUG o.a.c.c.file.GenericFileProducer - Wrote [D:\temp\GT_aplist_ghiiwb02_apache_20140111.lst] to [Endpoint[file://D:/temp]]
13:39:40.609 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] DEBUG o.a.c.c.file.GenericFileOnCompletion - Done processing file: RemoteFile[GT_aplist_ghiiwb02_apache_20140111.lst] using exchange: Exchange[GT_aplist_ghiiwb02_apache_20140111.lst]
13:39:40.781 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] WARN  o.a.c.c.file.remote.FtpConsumer - Consumer FtpConsumer[ftp://admin@localhost:2121/MF?doneFileName=%24%7Bfile%3Aname.noext%7D.TRG&maxMessagesPerPoll=10000&noop=true&passiveMode=true&password=xxxxxx] failed polling endpoint: Endpoint[ftp://admin@localhost:2121/MF?doneFileName=%24%7Bfile%3Aname.noext%7D.TRG&maxMessagesPerPoll=10000&noop=true&passiveMode=true&password=xxxxxx]. Will try again at next poll. Caused by: [java.lang.OutOfMemoryError - Java heap space]
java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2786) ~[na:1.6.0_13]
        at java.io.ByteArrayOutputStream.write(ByteArrayOutputStream.java:94) ~[na:1.6.0_13]
        at org.apache.commons.net.io.Util.copyStream(Util.java:119) ~[commons-net-2.0.jar:2.0]
        at org.apache.commons.net.ftp.FTPClient.retrieveFile(FTPClient.java:1286) ~[commons-net-2.0.jar:2.0]
        at org.apache.camel.component.file.remote.FtpOperations.retrieveFileToStreamInBody(FtpOperations.java:353) ~[camel-ftp-2.12.1.jar:2.12.1]
        at org.apache.camel.component.file.remote.FtpOperations.retrieveFile(FtpOperations.java:301) ~[camel-ftp-2.12.1.jar:2.12.1]
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:362) ~[camel-core-2.12.1.jar:2.12.1]
        at org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:99) ~[camel-ftp-2.12.1.jar:2.12.1]
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:201) ~[camel-core-2.12.1.jar:2.12.1]
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:165) ~[camel-core-2.12.1.jar:2.12.1]
        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) [camel-core-2.12.1.jar:2.12.1]
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:114) [camel-core-2.12.1.jar:2.12.1]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) [na:1.6.0_13]
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317) [na:1.6.0_13]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150) [na:1.6.0_13]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98) [na:1.6.0_13]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:181) [na:1.6.0_13]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:205) [na:1.6.0_13]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_13]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_13]
        at java.lang.Thread.run(Thread.java:619) [na:1.6.0_13]
13:39:41.296 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF
13:39:41.811 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF
13:39:42.326 [Camel (camel-1) thread #0 - ftp://admin@localhost:2121/MF] DEBUG o.a.c.c.file.remote.FtpConsumer - Took 0.000 seconds to poll: MF
Reply | Threaded
Open this post in threaded view
|

Re: out of heap space if remote FTP site has too many files to pick up

Claus Ibsen-2
In reply to this post by skasish
You can limit the number of files to pickup in the poll,
http://camel.apache.org/file2

See the maxMessagesPerPoll option

On Fri, Feb 21, 2014 at 9:13 AM, skasish <[hidden email]> wrote:

> out of heap space if remote FTP site has too many files to pick up.
>
> Is there configuration problem.
>
> My simple code is given below
>
> import org.slf4j.*;
> import org.apache.camel.*;
> import org.apache.camel.builder.*;
> import org.apache.camel.main.Main;
> import org.apache.camel.component.dataset.*;
>
> public class FileDemoCamel extends Main {
>     static Logger LOG = LoggerFactory.getLogger(FileDemoCamel.class);
>     public static void main(String[] args) throws Exception {
>         FileDemoCamel main = new FileDemoCamel();
>         main.enableHangupSupport();
>         main.addRouteBuilder(createRouteBuilder());
>         main.bind("sampleGenerator", createDataSet());
>         main.run(args);
>     }
>     static RouteBuilder createRouteBuilder() {
>         return new RouteBuilder() {
>             public void configure() {
>
> from("ftp://admin@localhost:2121/MF?noop=true&password=admin&passiveMode=true&maxMessagesPerPoll=10000&doneFileName=${file:name.noext}.TRG")
>                 .to("file://D:/temp");
>             }
>         };
>     }
>     static DataSet createDataSet() {
>         return new SimpleDataSet();
>     }
> }
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/out-of-heap-space-if-remote-FTP-site-has-too-many-files-to-pick-up-tp5747716.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io