Re: sFTP with multiple consumers and idempotent routes: file exceptions

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: sFTP with multiple consumers and idempotent routes: file exceptions

Claus Ibsen-2
Hi

The changed read-lock is not for competing consumers. But its for "the
file is currently being written and is not ready for any consumer to
download the file .. problem".

For competing consumers across nodes you need a read lock and
idempotent repository that works for these uses cases. If you have a
copy of the CiA2 book then this is covered in the clustering chapter.

On Fri, Jan 10, 2020 at 2:23 AM Mark Harris
<[hidden email]> wrote:

>
> Hello,
>
> I'm using Apache Camel version 2.24.2.
>
> I a Camel route that is processing files located an sFTP server. There are multiple nodes running the application where the route is located, so I have added file locking in an attempt to ensure only a single node process each file.
>
> The route URI looks like this:
> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=changed&readLockMinAge=10s&delete=true&delay=60s&include=<file mask>
>
> The route looks like this:
> from(inUri).id("myId")
>   .idempotentConsumer(header(Exchange.FILE_NAME), messageIdRepository).eager(true).skipDuplicate(true)
>   .log(LoggingLevel.INFO, "Processing file:  ${file:name}")
>   // Save original file in archive directory.
>   .to(archiveUri)
>   ... do other stuff...
>
> Every now and then, I'm getting what looks like file contention warning messages:
>
> Error processing file RemoteFile[<file>] due to
>   Cannot retrieve file: <file>. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: <file>]
>   org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: <file>
>     Caused by: com.jcraft.jsch.SftpException: No such file
>
> ... and also these:
>
> Error during commit. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot delete file: <file>]
>   org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file: <file>
>     Caused by: com.jcraft.jsch.SftpException: No such file
>
> Have I missed anything in my setup?
>
> I've also tried adding the idempotent repository to the input URI, like this:
> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=changed&readLockMinAge=<minAge>&delete=true&delay=<delay>&include=<file mask>&idempotent=true&idempotentKey=$simple{file:name}&idempotentRepository=#messageIdRepository
>
> ... but am getting the same kind of errors.
>
> Thanks for any assistance,
>
> Mark



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

RE: sFTP with multiple consumers and idempotent routes: file exceptions

Untitled
Hello,

I've had a look at the chapter in question, but it doesn't seem to cover my scenario of multiple consumers accessing an sFTP resource. Nor is it in the "Transactions and idempotency" chapter (chapter 12).

My idempotent repository is a JDBC repository, by the way, located on an Oracle database.

Looking at the Camel documentation:

https://camel.apache.org/components/latest/sftp-component.html

... it states that readLock=idempotent is only applicable for the file component, not the sFTP one.

Am I missing something?

Thanks,

Mark

-----Original Message-----
From: Claus Ibsen <[hidden email]>
Sent: 10 January 2020 17:24
To: [hidden email]
Subject: Re: sFTP with multiple consumers and idempotent routes: file exceptions

Hi

The changed read-lock is not for competing consumers. But its for "the file is currently being written and is not ready for any consumer to download the file .. problem".

For competing consumers across nodes you need a read lock and idempotent repository that works for these uses cases. If you have a copy of the CiA2 book then this is covered in the clustering chapter.

On Fri, Jan 10, 2020 at 2:23 AM Mark Harris <[hidden email]> wrote:

>
> Hello,
>
> I'm using Apache Camel version 2.24.2.
>
> I a Camel route that is processing files located an sFTP server. There are multiple nodes running the application where the route is located, so I have added file locking in an attempt to ensure only a single node process each file.
>
> The route URI looks like this:
> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=ch
> anged&readLockMinAge=10s&delete=true&delay=60s&include=<file mask>
>
> The route looks like this:
> from(inUri).id("myId")
>   .idempotentConsumer(header(Exchange.FILE_NAME), messageIdRepository).eager(true).skipDuplicate(true)
>   .log(LoggingLevel.INFO, "Processing file:  ${file:name}")
>   // Save original file in archive directory.
>   .to(archiveUri)
>   ... do other stuff...
>
> Every now and then, I'm getting what looks like file contention warning messages:
>
> Error processing file RemoteFile[<file>] due to
>   Cannot retrieve file: <file>. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: <file>]
>   org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: <file>
>     Caused by: com.jcraft.jsch.SftpException: No such file
>
> ... and also these:
>
> Error during commit. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot delete file: <file>]
>   org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file: <file>
>     Caused by: com.jcraft.jsch.SftpException: No such file
>
> Have I missed anything in my setup?
>
> I've also tried adding the idempotent repository to the input URI, like this:
> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=ch
> anged&readLockMinAge=<minAge>&delete=true&delay=<delay>&include=<file
> mask>&idempotent=true&idempotentKey=$simple{file:name}&idempotentRepos
> itory=#messageIdRepository
>
> ... but am getting the same kind of errors.
>
> Thanks for any assistance,
>
> Mark



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: sFTP with multiple consumers and idempotent routes: file exceptions

Jan Bednář
Hi Mark,
For JDBC idempotent repository is important to set primary key on
CAMEL_MESSAGEPROCESSED table, as stated in docs
https://camel.apache.org/components/latest/sql-component.html

"""
When working with concurrent consumers it is crucial to create a unique
constraint on the columns processorName and messageId. Because the
syntax for this constraint differs from database to database, we do not
show it here.
"""

Dne 12.1.2020 v 20:02 Mark Harris napsal(a):

> Hello,
>
> I've had a look at the chapter in question, but it doesn't seem to cover my scenario of multiple consumers accessing an sFTP resource. Nor is it in the "Transactions and idempotency" chapter (chapter 12).
>
> My idempotent repository is a JDBC repository, by the way, located on an Oracle database.
>
> Looking at the Camel documentation:
>
> https://camel.apache.org/components/latest/sftp-component.html
>
> ... it states that readLock=idempotent is only applicable for the file component, not the sFTP one.
>
> Am I missing something?
>
> Thanks,
>
> Mark
>
> -----Original Message-----
> From: Claus Ibsen <[hidden email]>
> Sent: 10 January 2020 17:24
> To: [hidden email]
> Subject: Re: sFTP with multiple consumers and idempotent routes: file exceptions
>
> Hi
>
> The changed read-lock is not for competing consumers. But its for "the file is currently being written and is not ready for any consumer to download the file .. problem".
>
> For competing consumers across nodes you need a read lock and idempotent repository that works for these uses cases. If you have a copy of the CiA2 book then this is covered in the clustering chapter.
>
> On Fri, Jan 10, 2020 at 2:23 AM Mark Harris <[hidden email]> wrote:
>> Hello,
>>
>> I'm using Apache Camel version 2.24.2.
>>
>> I a Camel route that is processing files located an sFTP server. There are multiple nodes running the application where the route is located, so I have added file locking in an attempt to ensure only a single node process each file.
>>
>> The route URI looks like this:
>> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=ch
>> anged&readLockMinAge=10s&delete=true&delay=60s&include=<file mask>
>>
>> The route looks like this:
>> from(inUri).id("myId")
>>    .idempotentConsumer(header(Exchange.FILE_NAME), messageIdRepository).eager(true).skipDuplicate(true)
>>    .log(LoggingLevel.INFO, "Processing file:  ${file:name}")
>>    // Save original file in archive directory.
>>    .to(archiveUri)
>>    ... do other stuff...
>>
>> Every now and then, I'm getting what looks like file contention warning messages:
>>
>> Error processing file RemoteFile[<file>] due to
>>    Cannot retrieve file: <file>. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: <file>]
>>    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: <file>
>>      Caused by: com.jcraft.jsch.SftpException: No such file
>>
>> ... and also these:
>>
>> Error during commit. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot delete file: <file>]
>>    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file: <file>
>>      Caused by: com.jcraft.jsch.SftpException: No such file
>>
>> Have I missed anything in my setup?
>>
>> I've also tried adding the idempotent repository to the input URI, like this:
>> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=ch
>> anged&readLockMinAge=<minAge>&delete=true&delay=<delay>&include=<file
>> mask>&idempotent=true&idempotentKey=$simple{file:name}&idempotentRepos
>> itory=#messageIdRepository
>>
>> ... but am getting the same kind of errors.
>>
>> Thanks for any assistance,
>>
>> Mark
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2


Reply | Threaded
Open this post in threaded view
|

RE: sFTP with multiple consumers and idempotent routes: file exceptions

Untitled
Hello,

Thanks for this. I have added a unique key to the processor name / message id columns.

I have also added ignoreFileNotFoundOrPermissionError=true to my route URI, yet I'm still getting errors:

Error during commit. Exchange[ID-vm-0-1578884164651-0-1]. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot delete file: <file>]
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file: <file>
        at org.apache.camel.component.file.remote.SftpOperations.deleteFile(SftpOperations.java:484) ~[camel-ftp-2.24.2.jar!/:2.24.2]
        ...
        at java.lang.Thread.run(Thread.java:748) [?:1.8.0_161]
Caused by: com.jcraft.jsch.SftpException: No such file
        at com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2873) ~[jsch-0.1.55.jar!/:?]
        at com.jcraft.jsch.ChannelSftp.rm(ChannelSftp.java:1985) ~[jsch-0.1.55.jar!/:?]
        at org.apache.camel.component.file.remote.SftpOperations.deleteFile(SftpOperations.java:480) ~[camel-ftp-2.24.2.jar!/:2.24.2]
        ... 24 more

Mark

-----Original Message-----
From: Jan Bednář <[hidden email]>
Sent: 13 January 2020 09:19
To: [hidden email]
Subject: Re: sFTP with multiple consumers and idempotent routes: file exceptions

Hi Mark,
For JDBC idempotent repository is important to set primary key on CAMEL_MESSAGEPROCESSED table, as stated in docs https://camel.apache.org/components/latest/sql-component.html

"""
When working with concurrent consumers it is crucial to create a unique constraint on the columns processorName and messageId. Because the syntax for this constraint differs from database to database, we do not show it here.
"""

Dne 12.1.2020 v 20:02 Mark Harris napsal(a):

> Hello,
>
> I've had a look at the chapter in question, but it doesn't seem to cover my scenario of multiple consumers accessing an sFTP resource. Nor is it in the "Transactions and idempotency" chapter (chapter 12).
>
> My idempotent repository is a JDBC repository, by the way, located on an Oracle database.
>
> Looking at the Camel documentation:
>
> https://camel.apache.org/components/latest/sftp-component.html
>
> ... it states that readLock=idempotent is only applicable for the file component, not the sFTP one.
>
> Am I missing something?
>
> Thanks,
>
> Mark
>
> -----Original Message-----
> From: Claus Ibsen <[hidden email]>
> Sent: 10 January 2020 17:24
> To: [hidden email]
> Subject: Re: sFTP with multiple consumers and idempotent routes: file
> exceptions
>
> Hi
>
> The changed read-lock is not for competing consumers. But its for "the file is currently being written and is not ready for any consumer to download the file .. problem".
>
> For competing consumers across nodes you need a read lock and idempotent repository that works for these uses cases. If you have a copy of the CiA2 book then this is covered in the clustering chapter.
>
> On Fri, Jan 10, 2020 at 2:23 AM Mark Harris <[hidden email]> wrote:
>> Hello,
>>
>> I'm using Apache Camel version 2.24.2.
>>
>> I a Camel route that is processing files located an sFTP server. There are multiple nodes running the application where the route is located, so I have added file locking in an attempt to ensure only a single node process each file.
>>
>> The route URI looks like this:
>> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=c
>> h anged&readLockMinAge=10s&delete=true&delay=60s&include=<file mask>
>>
>> The route looks like this:
>> from(inUri).id("myId")
>>    .idempotentConsumer(header(Exchange.FILE_NAME), messageIdRepository).eager(true).skipDuplicate(true)
>>    .log(LoggingLevel.INFO, "Processing file:  ${file:name}")
>>    // Save original file in archive directory.
>>    .to(archiveUri)
>>    ... do other stuff...
>>
>> Every now and then, I'm getting what looks like file contention warning messages:
>>
>> Error processing file RemoteFile[<file>] due to
>>    Cannot retrieve file: <file>. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot retrieve file: <file>]
>>    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot retrieve file: <file>
>>      Caused by: com.jcraft.jsch.SftpException: No such file
>>
>> ... and also these:
>>
>> Error during commit. Caused by: [org.apache.camel.component.file.GenericFileOperationFailedException - Cannot delete file: <file>]
>>    org.apache.camel.component.file.GenericFileOperationFailedException: Cannot delete file: <file>
>>      Caused by: com.jcraft.jsch.SftpException: No such file
>>
>> Have I missed anything in my setup?
>>
>> I've also tried adding the idempotent repository to the input URI, like this:
>> sftp://<user>@<server>:<port>/<folder>?password=<password>&readLock=c
>> h
>> anged&readLockMinAge=<minAge>&delete=true&delay=<delay>&include=<file
>> mask>&idempotent=true&idempotentKey=$simple{file:name}&idempotentRepo
>> mask>s
>> itory=#messageIdRepository
>>
>> ... but am getting the same kind of errors.
>>
>> Thanks for any assistance,
>>
>> Mark
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2