Cannot acquire read lock within x millis

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Cannot acquire read lock within x millis

Untitled
Hello,

I'm encountering the following message in my logs when two instances of a Java application running Camel on separate servers (to ensure HA) are trying to poll an sFTP server to process a file:

Cannot acquire read lock within 20000 millis. Will skip the file: RemoteFile[name and path of file on sFTP server]

The route URI is like this:

sftp://<user>@<server>:<port>/<path>?connectTimeout=120000&delay=60s&delete=true&disconnect=true&ignoreFileNotFoundOrPermissionError=true&inProgressRepository=<repository>&include=<mask>&password=<password>&readLock=changed&readLockMinAge=10s&soTimeout=300000

The process copies the file to a local shared folder where it is further processed by routes running on the same 2 Java applications with a URI like this:

file://<path>?delay=60s&inProgressRepository=<repository>&include=<mask>&readLock=changed&readLockMinAge=10s

... where I also get the same message:

Cannot acquire read lock within 10000 millis. Will skip the file: GenericFile[name and path of file on shared folder]

Eventually the file is processed but it is taking several polls before one of the applications processes the file, leading to delays.

Can anyone advise what the issue is and how I can fix it, please?

I'm using Camel v 2.24.2.

Thanks,

Mark

Reply | Threaded
Open this post in threaded view
|

RE: Cannot acquire read lock within x millis

Untitled
I should add that I'm seeing the "Cannot acquire read lock within x millis" messages appear in the logs of both application servers several seconds apart, implying that the two instances are not polling at precisely the same time. Yet neither of them are able to process the file for several rounds of polling before one of them does.

Thanks,

Mark

From: Mark Harris
Sent: 18 March 2020 13:55
To: [hidden email]
Subject: Cannot acquire read lock within x millis

Hello,

I'm encountering the following message in my logs when two instances of a Java application running Camel on separate servers (to ensure HA) are trying to poll an sFTP server to process a file:

Cannot acquire read lock within 20000 millis. Will skip the file: RemoteFile[name and path of file on sFTP server]

The route URI is like this:

sftp://<user>@<server>:<port>/<path>?connectTimeout=120000&delay=60s&delete=true&disconnect=true&ignoreFileNotFoundOrPermissionError=true&inProgressRepository=<repository>&include=<mask>&password=<password>&readLock=changed&readLockMinAge=10s&soTimeout=300000

The process copies the file to a local shared folder where it is further processed by routes running on the same 2 Java applications with a URI like this:

file://<path>?delay=60s&inProgressRepository=<repository>&include=<mask>&readLock=changed&readLockMinAge=10s<file://%3cpath%3e?delay=60s&inProgressRepository=%3crepository%3e&include=%3cmask%3e&readLock=changed&readLockMinAge=10s>

... where I also get the same message:

Cannot acquire read lock within 10000 millis. Will skip the file: GenericFile[name and path of file on shared folder]

Eventually the file is processed but it is taking several polls before one of the applications processes the file, leading to delays.

Can anyone advise what the issue is and how I can fix it, please?

I'm using Camel v 2.24.2.

Thanks,

Mark

Reply | Threaded
Open this post in threaded view
|

RE: Cannot acquire read lock within x millis

Untitled
Hello,

Can anyone suggest what the problem is here, please? It sounds like a pretty fundamental issue but I have not managed to find any similar reports (and potential solutions) elsewhere.

Thanks,

Mark

From: Mark Harris
Sent: 18 March 2020 14:51
To: [hidden email]
Subject: RE: Cannot acquire read lock within x millis

I should add that I'm seeing the "Cannot acquire read lock within x millis" messages appear in the logs of both application servers several seconds apart, implying that the two instances are not polling at precisely the same time. Yet neither of them are able to process the file for several rounds of polling before one of them does.

Thanks,

Mark

From: Mark Harris
Sent: 18 March 2020 13:55
To: [hidden email]<mailto:[hidden email]>
Subject: Cannot acquire read lock within x millis

Hello,

I'm encountering the following message in my logs when two instances of a Java application running Camel on separate servers (to ensure HA) are trying to poll an sFTP server to process a file:

Cannot acquire read lock within 20000 millis. Will skip the file: RemoteFile[name and path of file on sFTP server]

The route URI is like this:

sftp://<user>@<server>:<port>/<path>?connectTimeout=120000&delay=60s&delete=true&disconnect=true&ignoreFileNotFoundOrPermissionError=true&inProgressRepository=<repository>&include=<mask>&password=<password>&readLock=changed&readLockMinAge=10s&soTimeout=300000

The process copies the file to a local shared folder where it is further processed by routes running on the same 2 Java applications with a URI like this:

file://<path>?delay=60s&inProgressRepository=<repository>&include=<mask>&readLock=changed&readLockMinAge=10s<file://%3cpath%3e?delay=60s&inProgressRepository=%3crepository%3e&include=%3cmask%3e&readLock=changed&readLockMinAge=10s>

... where I also get the same message:

Cannot acquire read lock within 10000 millis. Will skip the file: GenericFile[name and path of file on shared folder]

Eventually the file is processed but it is taking several polls before one of the applications processes the file, leading to delays.

Can anyone advise what the issue is and how I can fix it, please?

I'm using Camel v 2.24.2.

Thanks,

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Cannot acquire read lock within x millis

Claus Ibsen-2
Hi

Its been talked about before, that changed read-locks on FTP servers
is not "so fast" because FTP server often returns file information /
file lists with only hh:mm information and not seconds, so a change
strategy cannot use seconds to know if a file was changed or not.

On Thu, Mar 19, 2020 at 12:57 AM Mark Harris
<[hidden email]> wrote:

>
> Hello,
>
> Can anyone suggest what the problem is here, please? It sounds like a pretty fundamental issue but I have not managed to find any similar reports (and potential solutions) elsewhere.
>
> Thanks,
>
> Mark
>
> From: Mark Harris
> Sent: 18 March 2020 14:51
> To: [hidden email]
> Subject: RE: Cannot acquire read lock within x millis
>
> I should add that I'm seeing the "Cannot acquire read lock within x millis" messages appear in the logs of both application servers several seconds apart, implying that the two instances are not polling at precisely the same time. Yet neither of them are able to process the file for several rounds of polling before one of them does.
>
> Thanks,
>
> Mark
>
> From: Mark Harris
> Sent: 18 March 2020 13:55
> To: [hidden email]<mailto:[hidden email]>
> Subject: Cannot acquire read lock within x millis
>
> Hello,
>
> I'm encountering the following message in my logs when two instances of a Java application running Camel on separate servers (to ensure HA) are trying to poll an sFTP server to process a file:
>
> Cannot acquire read lock within 20000 millis. Will skip the file: RemoteFile[name and path of file on sFTP server]
>
> The route URI is like this:
>
> sftp://<user>@<server>:<port>/<path>?connectTimeout=120000&delay=60s&delete=true&disconnect=true&ignoreFileNotFoundOrPermissionError=true&inProgressRepository=<repository>&include=<mask>&password=<password>&readLock=changed&readLockMinAge=10s&soTimeout=300000
>
> The process copies the file to a local shared folder where it is further processed by routes running on the same 2 Java applications with a URI like this:
>
> file://<path>?delay=60s&inProgressRepository=<repository>&include=<mask>&readLock=changed&readLockMinAge=10s<file://%3cpath%3e?delay=60s&inProgressRepository=%3crepository%3e&include=%3cmask%3e&readLock=changed&readLockMinAge=10s>
>
> ... where I also get the same message:
>
> Cannot acquire read lock within 10000 millis. Will skip the file: GenericFile[name and path of file on shared folder]
>
> Eventually the file is processed but it is taking several polls before one of the applications processes the file, leading to delays.
>
> Can anyone advise what the issue is and how I can fix it, please?
>
> I'm using Camel v 2.24.2.
>
> Thanks,
>
> Mark
>


--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

RE: Cannot acquire read lock within x millis

Untitled
> Its been talked about before, that changed read-locks on FTP servers is not "so fast" because FTP server
> often returns file information / file lists with only hh:mm information and not seconds, so a change
> strategy cannot use seconds to know if a file was changed or not.

Is there an alternative strategy that I can use?

The scenario is that I have two applications polling an sFTP server for new files every 30 seconds; any new file needs to be at least 10 seconds old, picked up as soon as possible after this period and processed by only one of the applications.

I'm currently using readLock=changed, a readLockMinAge and an inProgressRepository to ensure idempotency in my route URI.

Thanks,

Mark
Reply | Threaded
Open this post in threaded view
|

Re: Cannot acquire read lock within x millis

Claus Ibsen-2
Hi

Its better if when uploading the files they are uploaded using a
temporary name and then renamed when done.
eg if the network fails during upload then you know the file is not
complete. If you dont do this, you can essentially have broken files.

However if you know that a file is complete if its been there for 10
seconds, then you can write your own custom read lock strategy that
keeps track of this and then returns true in those situations.


On Fri, Mar 20, 2020 at 1:17 AM Mark Harris
<[hidden email]> wrote:

>
> > Its been talked about before, that changed read-locks on FTP servers is not "so fast" because FTP server
> > often returns file information / file lists with only hh:mm information and not seconds, so a change
> > strategy cannot use seconds to know if a file was changed or not.
>
> Is there an alternative strategy that I can use?
>
> The scenario is that I have two applications polling an sFTP server for new files every 30 seconds; any new file needs to be at least 10 seconds old, picked up as soon as possible after this period and processed by only one of the applications.
>
> I'm currently using readLock=changed, a readLockMinAge and an inProgressRepository to ensure idempotency in my route URI.
>
> Thanks,
>
> Mark



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2