Quantcast

Camel processes only the first 150 rows in CSV

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Camel processes only the first 150 rows in CSV

Ambika Das-2
Hi All,


I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.


my code looks as follows.


@override

public void configure() {

String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"

from(endpoint).to("direct:file_Processor")


}


Regards,

Prasad
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel processes only the first 150 rows in CSV

Ambika Das-2
I put the file in the SFTP drop box and started the application. It takes up the first 150 rows for processing.


Any reason why Camel skips rest of the rows?

Regards,

Prasad

________________________________
From: Ambika Das <[hidden email]>
Sent: Thursday, May 18, 2017 12:28 PM
To: [hidden email]
Subject: Camel processes only the first 150 rows in CSV

Hi All,


I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.


my code looks as follows.


@override

public void configure() {

String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"

from(endpoint).to("direct:file_Processor")


}


Regards,

Prasad
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel processes only the first 150 rows in CSV

Zoran Regvart-2
In reply to this post by Ambika Das-2
Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.

Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.

The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.

zoran

zoran

On Thu, May 18, 2017 at 8:58 AM, Ambika Das <[hidden email]> wrote:

> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel processes only the first 150 rows in CSV

Ambika Das-2
I changed the code and my end point looks like this.


"sftp://server:port/dropbox?delay=10000&doneFileName=done&idempotent=true&moveFailed=failed&noop=true&password=xxxxxx&preMove=done&readLock=true&username=sftpuser1"


It is still stopping at 150 rows.


Regards,

Prasad


________________________________
From: Zoran Regvart <[hidden email]>
Sent: Thursday, May 18, 2017 1:10 PM
To: [hidden email]
Subject: Re: Camel processes only the first 150 rows in CSV

Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.

Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.

The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.

zoran

zoran

On Thu, May 18, 2017 at 8:58 AM, Ambika Das <[hidden email]> wrote:

> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel processes only the first 150 rows in CSV

Ambika Das-2
I am using currently readLock and doneFileName, but the issue persists.


sftp://server:port/data?delay=10000&doneFileName=${file:name}.done&move=.done&moveFailed=failed&noop=true&password=xxxxxx&readLock=changed&username=abcd


Regards,

Ambika


________________________________
From: Ambika Das <[hidden email]>
Sent: Thursday, May 18, 2017 1:50 PM
To: [hidden email]
Subject: Re: Camel processes only the first 150 rows in CSV

I changed the code and my end point looks like this.


"sftp://server:port/dropbox?delay=10000&doneFileName=done&idempotent=true&moveFailed=failed&noop=true&password=xxxxxx&preMove=done&readLock=true&username=sftpuser1"


It is still stopping at 150 rows.


Regards,

Prasad


________________________________
From: Zoran Regvart <[hidden email]>
Sent: Thursday, May 18, 2017 1:10 PM
To: [hidden email]
Subject: Re: Camel processes only the first 150 rows in CSV

Hi,
you have specified `noop=true` and I guess you didn't configure
`idempotent=true` and `idempotentRepository` (if you wish to preserve
the state of processing between nodes and restarts) -- that keeps the
file as is on the SFTP server.

Also the file might be half-consumed if you're not using
`doneFileName`. It works like this: Camel notices a file on the SFTP
server and starts processing it while the app that is placing the file
is still writing to the file. It could be just that the app that is
writing the file in the first place has a buffer, so on first buffer
flush only first 150 lines are written.

The best course of action is to combine `readLock` and `idempotent*`
settings with `doneFileName`, i.e. the app that is writing the file to
the SFTP server leaves another file (a marker file) that indicates
that it's done writing.

zoran

zoran

On Thu, May 18, 2017 at 8:58 AM, Ambika Das <[hidden email]> wrote:

> Hi All,
>
>
> I am developing a aggregator application which is supposed to consume a CSV and generate the message. I have developed it as a Spring boot application. The problem is when I transfer a file it takes only the first 150 record and processes. The file is still there in the folder. If I restart the application the whole file gets processed.
>
>
> my code looks as follows.
>
>
> @override
>
> public void configure() {
>
> String end point = "sftp://server:port/dropbox?delay=10000&moveFailed=failed&noop=true&preMove=done"
>
> from(endpoint).to("direct:file_Processor")
>
>
> }
>
>
> Regards,
>
> Prasad



--
Zoran Regvart
Loading...