File Component route issue.

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
jay
Reply | Threaded
Open this post in threaded view
|

File Component route issue.

jay
Hi I have a requirement to process 2 huge files from remote server.

Step1:i have written route to SFTP the file from remote server to local server - working fine.

from(from).autoStartup(true).to(to)

from--> sftp://user@xxxx?password=xxxx&localWorkDirectory=/tmp&readLock=changed&idempotent=true&move=processed
to--> file:///opt/mydata

Step2: consuming the file from local directory and splitting that and sending to queue.

from(from)
        .routeId("split route")
                .split(body().tokenize("\n")).streaming().convertBodyTo(String.class)
                .process(new Processor())
                .to(splitQueue);

from-->file:///opt/mydata?move=../processed&readLock=changed&idempotent=true
splitQueue--->activemq:splitqueue


I am running the above in Fuse ESB  and is working OK in Macbook -  but when i deploy the bundle in linux server - first file (small file 3500 records) went through OK but when it process the second file Step2 route is keep on reading the huge file (6million records) and keep on sending it to queue with out stopping (i have noticed it read only 3000 rows and keep on sending the same 3000 rows).


exception:
21:44:26,537 | ERROR | /opt/mydata | DefaultErrorHandler              | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Failed delivery for (MessageId: ID-ABC-41834-1392325070932-5-167921 on ExchangeId: ID-ABC-41834-1392325070932-5-167922). Exhausted after delivery attempt: 1 caught: org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!
org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!
        at org.apache.camel.processor.Splitter$1$1.hasNext(Splitter.java:156)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]

Caused by: java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:260)[:1.6.0_45]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)[:1.6.0_45]

can some one help me please as it is really stopper issue for my delivery?
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

Claus Ibsen-2
Hi

What about the line separators on the big file? Are it just \n or is
it maybe \n\r or \r etc?
Maybe the tokenizer need to be adjusted?

You can also try a little java main app and try to use the java util
Scanner to read the same file and see if you can do that without any
issues. And is there something special in about row 3000 where it
"seems to stop" ?

On Thu, Feb 13, 2014 at 11:53 PM, jay <[hidden email]> wrote:

> Hi I have a requirement to process 2 huge files from remote server.
>
> Step1:i have written route to SFTP the file from remote server to local
> server - working fine.
>
> from(from).autoStartup(true).to(to)
>
> from-->
> sftp://user@xxxx?password=xxxx&localWorkDirectory=/tmp&readLock=changed&idempotent=true&move=processed
> to--> file:///opt/mydata
>
> Step2: consuming the file from local directory and splitting that and
> sending to queue.
>
> from(from)
>         .routeId("split route")
>
> .split(body().tokenize("\n")).streaming().convertBodyTo(String.class)
>                 .process(new Processor())
>                 .to(splitQueue);
>
> from-->file:///opt/mydata?move=../processed&readLock=changed&idempotent=true
> splitQueue--->activemq:splitqueue
>
>
> I am running the above in Fuse ESB  and is working OK in Macbook -  but when
> i deploy the bundle in linux server - first file (small file 3500 records)
> went through OK but when it process the second file Step2 route is keep on
> reading the huge file (6million records) and keep on sending it to queue
> with out stopping (i have noticed it read only 3000 rows and keep on sending
> the same 3000 rows).
>
>
> exception:
> 21:44:26,537 | ERROR | /opt/mydata | DefaultErrorHandler              | 147
> - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Failed delivery for
> (MessageId: ID-ABC-41834-1392325070932-5-167921 on ExchangeId:
> ID-ABC-41834-1392325070932-5-167922). Exhausted after delivery attempt: 1
> caught: org.apache.camel.RuntimeCamelException: Scanner aborted because of
> an IOException!
> org.apache.camel.RuntimeCamelException: Scanner aborted because of an
> IOException!
>         at
> org.apache.camel.processor.Splitter$1$1.hasNext(Splitter.java:156)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
>
> Caused by: java.nio.charset.MalformedInputException: Input length = 1
>         at
> java.nio.charset.CoderResult.throwException(CoderResult.java:260)[:1.6.0_45]
>         at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)[:1.6.0_45]
>
> can some one help me please as it is really stopper issue for my delivery?
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Component-route-issue-tp5747281.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
jay
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

jay
Thanks Claus - thanks for your views. i have tried with tokenizer locally and it processed all the records.
 
however i  think tokenizer may not be a problem as the same file has been processed in my local fuse container (Macbook)- i have  tried couple of times in local fuse esb and it processed.

I am suspecting may be some cache/memory/aggregation problem in the container as it is keep on the reading only few records and processing them again and again.



jay
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

jay
This post was updated on .
In reply to this post by Claus Ibsen-2
Hi Claus,
it is both as we might get the CR LF or LF in file so i used both.  And also you are right that the tokenizer needs to be adjusted ignoring the invalid lines/chars - looks like it is not a memory issue - it is issue with the program.

 
.split(body().tokenize("\n|\r\n")).streaming().convertBodyTo(String.class)
                .process(some Processor)
                .to(splitMessageQueue);


Error is  - exhausted after delivery attempt: 1 caught: org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!

would you mind to provide me your views on fixing the issue.


full stack trace:
org.apache.camel.RuntimeCamelException: Scanner aborted because of an IOException!
        at org.apache.camel.processor.Splitter$1$1.hasNext(Splitter.java:156)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.Splitter.updateNewExchange(Splitter.java:213)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:502)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.MulticastProcessor.process(MulticastProcessor.java:213)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.Splitter.process(Splitter.java:98)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:91)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.fabric.FabricTraceProcessor.process(FabricTraceProcessor.java:81)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:334)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:220)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.interceptor.StreamCachingInterceptor.process(StreamCachingInterceptor.java:52)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.interceptor.DefaultChannel.process(DefaultChannel.java:303)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:117)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.Pipeline.process(Pipeline.java:80)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.RouteContextProcessor.processNext(RouteContextProcessor.java:45)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.UnitOfWorkProcessor.processAsync(UnitOfWorkProcessor.java:150)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:117)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.RouteInflightRepositoryProcessor.processNext(RouteInflightRepositoryProcessor.java:48)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:99)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:90)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:73)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:336)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:189)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:155)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:142)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:92)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)[:1.6.0_45]
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)[:1.6.0_45]
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)[:1.6.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)[:1.6.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)[:1.6.0_45]
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)[:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)[:1.6.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)[:1.6.0_45]
        at java.lang.Thread.run(Thread.java:662)[:1.6.0_45]
Caused by: java.nio.charset.MalformedInputException: Input length = 1
        at java.nio.charset.CoderResult.throwException(CoderResult.java:260)[:1.6.0_45]
        at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:319)[:1.6.0_45]
        at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)[:1.6.0_45]
        at java.io.Reader.read(Reader.java:83)[:1.6.0_45]
        at java.util.Scanner.readInput(Scanner.java:780)[:1.6.0_45]
        at java.util.Scanner.next(Scanner.java:1345)[:1.6.0_45]
        at java.util.Scanner.next(Scanner.java:328)[:1.6.0_45]
        at org.apache.camel.processor.Splitter$1$1.next(Splitter.java:164)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.Splitter$1$1.next(Splitter.java:133)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        at org.apache.camel.processor.MulticastProcessor.doProcessSequential(MulticastProcessor.java:500)[147:org.apache.camel.camel-core:2.10.0.fuse-71-047]
        ... 48 more
13:44:12,980 | WARN  | /monthly/inbound | GenericFileOnCompletion          | 147 - org.apache.camel.camel-core - 2.10.0.fuse-71-047 | Rollback file strategy: org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy@54f37d22 for file: GenericFile[/opt/data/interfaces/inbound/AAAA.TXT]
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

Claus Ibsen-2
Hi

I found this ticket, which refers to another thread
https://issues.apache.org/jira/browse/CAMEL-7100


On Wed, Feb 19, 2014 at 3:02 PM, jay <[hidden email]> wrote:

> Hi Claus,
> it is both as we might get the CR LF or LF in file so i used both.  And also
> you are right that the tokenizer needs to be adjusted ignoring the invalid
> lines/chars - looks like it is not a memory issue - it is issue with the
> program.
>
>
> .split(body().tokenize("\n|\r\n")).streaming().convertBodyTo(String.class)
>                 .process(some Processor)
>                 .to(splitMessageQueue);
>
>
> Error is  - exhausted after delivery attempt: 1 caught:
> org.apache.camel.RuntimeCamelException: Scanner aborted because of an
> IOException!
>
> would you mind to provide me your views on fixing the issue.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File-Component-route-issue-tp5747281p5747550.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
jay
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

jay
Hello Claus,
Do you suggest any work around for this issue?
Reply | Threaded
Open this post in threaded view
|

Re: File Component route issue.

Stéphane Chaillon
I had the same issue today.
It was simply linked to the file encoding.

I just had charset option to the file component and the problem has been solved !