Camel lock gets acquired on files but not picked sometime

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Camel lock gets acquired on files but not picked sometime

Hemant Pekhale
Hi All,

I have a route which picks files from a folder and do some processing on it
and store it in other folder. I have used parameter "delete=true" for
deleting file from source folder.

Sometime it happens that files are stuck in the source folder and camel
lock files are also created for all the files but files are neither process
nor gets deleted.

There is a VB script which copied files in the source folder. When I
manually copied the files in the source folder then it never gives issue
but when the files are copied from VB script then only issue occurs. Also
the issue does not occur for the first file but it occurs randomly after
processing number of files.

This happens on server. I tried to reproduce this issue locally but no luck.

Does anybody have idea about it?

Thanks!
Regards,
Hemant Pekhale
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Claus Ibsen-2
Hi

Are you able to modify the VB script so it copies the file using a
temporary name and then rename the file when the copy is done.

It sounds a bit like the problem is that the VB script is currently
still copying the file, while Camel gets a lock on the file, and
attempt to process it.
File locks is not as easy and it depends a bit what file system, OS,
and how big the files are and other things.

A good strategy is to copy the files using temporary names or in
another folder, and then do a file rename when the copy is over, that
latter then is an atomic file operation which should not cause file
locking issues.

You can also try to use some of the other Camel file lock strategies
instead of marker files.
You can see more in the file documentation about these.


On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <[hidden email]> wrote:

> Hi All,
>
> I have a route which picks files from a folder and do some processing on it
> and store it in other folder. I have used parameter "delete=true" for
> deleting file from source folder.
>
> Sometime it happens that files are stuck in the source folder and camel
> lock files are also created for all the files but files are neither process
> nor gets deleted.
>
> There is a VB script which copied files in the source folder. When I
> manually copied the files in the source folder then it never gives issue
> but when the files are copied from VB script then only issue occurs. Also
> the issue does not occur for the first file but it occurs randomly after
> processing number of files.
>
> This happens on server. I tried to reproduce this issue locally but no luck.
>
> Does anybody have idea about it?
>
> Thanks!



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Hemant Pekhale
Hi Claus,

Thanks for the reply.

Yes, I am able to modify the script. But if we do copy the file with
temporary name and then rename, I doubt that there may be lock acquired on
temporary file name already. because my route has no waiting parameter. So
I think it might not be helpful for our case. Please correct me if I am
wrong.

Regarding the file lock strategies, I think  readLock think
readLockCheckInterval think can be tried. Do you have any other suggestion?

FYI : Whatever I can try I need to try on Production server so just wanted
to tackle this with minimum and required trials.

Thanks!


On Mon, Aug 6, 2018 at 7:10 PM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> Are you able to modify the VB script so it copies the file using a
> temporary name and then rename the file when the copy is done.
>
> It sounds a bit like the problem is that the VB script is currently
> still copying the file, while Camel gets a lock on the file, and
> attempt to process it.
> File locks is not as easy and it depends a bit what file system, OS,
> and how big the files are and other things.
>
> A good strategy is to copy the files using temporary names or in
> another folder, and then do a file rename when the copy is over, that
> latter then is an atomic file operation which should not cause file
> locking issues.
>
> You can also try to use some of the other Camel file lock strategies
> instead of marker files.
> You can see more in the file documentation about these.
>
>
> On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <[hidden email]>
> wrote:
> > Hi All,
> >
> > I have a route which picks files from a folder and do some processing on
> it
> > and store it in other folder. I have used parameter "delete=true" for
> > deleting file from source folder.
> >
> > Sometime it happens that files are stuck in the source folder and camel
> > lock files are also created for all the files but files are neither
> process
> > nor gets deleted.
> >
> > There is a VB script which copied files in the source folder. When I
> > manually copied the files in the source folder then it never gives issue
> > but when the files are copied from VB script then only issue occurs. Also
> > the issue does not occur for the first file but it occurs randomly after
> > processing number of files.
> >
> > This happens on server. I tried to reproduce this issue locally but no
> luck.
> >
> > Does anybody have idea about it?
> >
> > Thanks!
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Regards,
Hemant Pekhale
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Claus Ibsen-2
Hi

You can configure Camel to only pickup files with a certain file
pattern, or to exclude names with temporary names, etc.
There is a number of different filter options, see its doc.

On Mon, Aug 6, 2018 at 3:52 PM, Hemant Pekhale <[hidden email]> wrote:

> Hi Claus,
>
> Thanks for the reply.
>
> Yes, I am able to modify the script. But if we do copy the file with
> temporary name and then rename, I doubt that there may be lock acquired on
> temporary file name already. because my route has no waiting parameter. So
> I think it might not be helpful for our case. Please correct me if I am
> wrong.
>
> Regarding the file lock strategies, I think  readLock think
> readLockCheckInterval think can be tried. Do you have any other suggestion?
>
> FYI : Whatever I can try I need to try on Production server so just wanted
> to tackle this with minimum and required trials.
>
> Thanks!
>
>
> On Mon, Aug 6, 2018 at 7:10 PM, Claus Ibsen <[hidden email]> wrote:
>
>> Hi
>>
>> Are you able to modify the VB script so it copies the file using a
>> temporary name and then rename the file when the copy is done.
>>
>> It sounds a bit like the problem is that the VB script is currently
>> still copying the file, while Camel gets a lock on the file, and
>> attempt to process it.
>> File locks is not as easy and it depends a bit what file system, OS,
>> and how big the files are and other things.
>>
>> A good strategy is to copy the files using temporary names or in
>> another folder, and then do a file rename when the copy is over, that
>> latter then is an atomic file operation which should not cause file
>> locking issues.
>>
>> You can also try to use some of the other Camel file lock strategies
>> instead of marker files.
>> You can see more in the file documentation about these.
>>
>>
>> On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <[hidden email]>
>> wrote:
>> > Hi All,
>> >
>> > I have a route which picks files from a folder and do some processing on
>> it
>> > and store it in other folder. I have used parameter "delete=true" for
>> > deleting file from source folder.
>> >
>> > Sometime it happens that files are stuck in the source folder and camel
>> > lock files are also created for all the files but files are neither
>> process
>> > nor gets deleted.
>> >
>> > There is a VB script which copied files in the source folder. When I
>> > manually copied the files in the source folder then it never gives issue
>> > but when the files are copied from VB script then only issue occurs. Also
>> > the issue does not occur for the first file but it occurs randomly after
>> > processing number of files.
>> >
>> > This happens on server. I tried to reproduce this issue locally but no
>> luck.
>> >
>> > Does anybody have idea about it?
>> >
>> > Thanks!
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> http://davsclaus.com @davsclaus
>> Camel in Action 2: https://www.manning.com/ibsen2
>>



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Claus Ibsen-2
Also you can use "done" files, eg in the VB script you write a 2nd
empty file with a special name, that tells Camel that the other file
is done being copied and ready.
There is some options on Camel file endpoint you can configure for that also.

On Mon, Aug 6, 2018 at 3:54 PM, Claus Ibsen <[hidden email]> wrote:

> Hi
>
> You can configure Camel to only pickup files with a certain file
> pattern, or to exclude names with temporary names, etc.
> There is a number of different filter options, see its doc.
>
> On Mon, Aug 6, 2018 at 3:52 PM, Hemant Pekhale <[hidden email]> wrote:
>> Hi Claus,
>>
>> Thanks for the reply.
>>
>> Yes, I am able to modify the script. But if we do copy the file with
>> temporary name and then rename, I doubt that there may be lock acquired on
>> temporary file name already. because my route has no waiting parameter. So
>> I think it might not be helpful for our case. Please correct me if I am
>> wrong.
>>
>> Regarding the file lock strategies, I think  readLock think
>> readLockCheckInterval think can be tried. Do you have any other suggestion?
>>
>> FYI : Whatever I can try I need to try on Production server so just wanted
>> to tackle this with minimum and required trials.
>>
>> Thanks!
>>
>>
>> On Mon, Aug 6, 2018 at 7:10 PM, Claus Ibsen <[hidden email]> wrote:
>>
>>> Hi
>>>
>>> Are you able to modify the VB script so it copies the file using a
>>> temporary name and then rename the file when the copy is done.
>>>
>>> It sounds a bit like the problem is that the VB script is currently
>>> still copying the file, while Camel gets a lock on the file, and
>>> attempt to process it.
>>> File locks is not as easy and it depends a bit what file system, OS,
>>> and how big the files are and other things.
>>>
>>> A good strategy is to copy the files using temporary names or in
>>> another folder, and then do a file rename when the copy is over, that
>>> latter then is an atomic file operation which should not cause file
>>> locking issues.
>>>
>>> You can also try to use some of the other Camel file lock strategies
>>> instead of marker files.
>>> You can see more in the file documentation about these.
>>>
>>>
>>> On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <[hidden email]>
>>> wrote:
>>> > Hi All,
>>> >
>>> > I have a route which picks files from a folder and do some processing on
>>> it
>>> > and store it in other folder. I have used parameter "delete=true" for
>>> > deleting file from source folder.
>>> >
>>> > Sometime it happens that files are stuck in the source folder and camel
>>> > lock files are also created for all the files but files are neither
>>> process
>>> > nor gets deleted.
>>> >
>>> > There is a VB script which copied files in the source folder. When I
>>> > manually copied the files in the source folder then it never gives issue
>>> > but when the files are copied from VB script then only issue occurs. Also
>>> > the issue does not occur for the first file but it occurs randomly after
>>> > processing number of files.
>>> >
>>> > This happens on server. I tried to reproduce this issue locally but no
>>> luck.
>>> >
>>> > Does anybody have idea about it?
>>> >
>>> > Thanks!
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> http://davsclaus.com @davsclaus
>>> Camel in Action 2: https://www.manning.com/ibsen2
>>>
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Trilok
You can make the processes mutually exclusive through setting up a
scheduler by giving a cron expression so that your FTP or File component
would not read files when they are being written, just in case
if processing of your file not real time and copying of file via VB script
is also scheduled.

Otherwise readLock=changed would help, try it out-
From camel documentation-

*changed* is using file length/modification timestamp to detect whether the
file is currently being copied or not. Will at least use 1 sec. to
determine this, so this option cannot consume files as fast as the others,
but can be more reliable as the JDK IO API cannot always determine whether
a file is currently being used by another process. The option
*readLockCheckInterval* can be used to set the check frequency. This option
is *only* avail for the FTP <http://camel.apache.org/ftp2.html> component
from *Camel 2.8* onward. Note: from *Camel 2.10.1* onward the FTP
<http://camel.apache.org/ftp2.html> option *fastExistsCheck* can be enabled
to speedup this *readLock* strategy, if the FTP server support the LIST
operation with a full file name (some servers may not)

On Mon, Aug 6, 2018 at 7:26 PM, Claus Ibsen <[hidden email]> wrote:

> Also you can use "done" files, eg in the VB script you write a 2nd
> empty file with a special name, that tells Camel that the other file
> is done being copied and ready.
> There is some options on Camel file endpoint you can configure for that
> also.
>
> On Mon, Aug 6, 2018 at 3:54 PM, Claus Ibsen <[hidden email]> wrote:
> > Hi
> >
> > You can configure Camel to only pickup files with a certain file
> > pattern, or to exclude names with temporary names, etc.
> > There is a number of different filter options, see its doc.
> >
> > On Mon, Aug 6, 2018 at 3:52 PM, Hemant Pekhale <[hidden email]>
> wrote:
> >> Hi Claus,
> >>
> >> Thanks for the reply.
> >>
> >> Yes, I am able to modify the script. But if we do copy the file with
> >> temporary name and then rename, I doubt that there may be lock acquired
> on
> >> temporary file name already. because my route has no waiting parameter.
> So
> >> I think it might not be helpful for our case. Please correct me if I am
> >> wrong.
> >>
> >> Regarding the file lock strategies, I think  readLock think
> >> readLockCheckInterval think can be tried. Do you have any other
> suggestion?
> >>
> >> FYI : Whatever I can try I need to try on Production server so just
> wanted
> >> to tackle this with minimum and required trials.
> >>
> >> Thanks!
> >>
> >>
> >> On Mon, Aug 6, 2018 at 7:10 PM, Claus Ibsen <[hidden email]>
> wrote:
> >>
> >>> Hi
> >>>
> >>> Are you able to modify the VB script so it copies the file using a
> >>> temporary name and then rename the file when the copy is done.
> >>>
> >>> It sounds a bit like the problem is that the VB script is currently
> >>> still copying the file, while Camel gets a lock on the file, and
> >>> attempt to process it.
> >>> File locks is not as easy and it depends a bit what file system, OS,
> >>> and how big the files are and other things.
> >>>
> >>> A good strategy is to copy the files using temporary names or in
> >>> another folder, and then do a file rename when the copy is over, that
> >>> latter then is an atomic file operation which should not cause file
> >>> locking issues.
> >>>
> >>> You can also try to use some of the other Camel file lock strategies
> >>> instead of marker files.
> >>> You can see more in the file documentation about these.
> >>>
> >>>
> >>> On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <
> [hidden email]>
> >>> wrote:
> >>> > Hi All,
> >>> >
> >>> > I have a route which picks files from a folder and do some
> processing on
> >>> it
> >>> > and store it in other folder. I have used parameter "delete=true" for
> >>> > deleting file from source folder.
> >>> >
> >>> > Sometime it happens that files are stuck in the source folder and
> camel
> >>> > lock files are also created for all the files but files are neither
> >>> process
> >>> > nor gets deleted.
> >>> >
> >>> > There is a VB script which copied files in the source folder. When I
> >>> > manually copied the files in the source folder then it never gives
> issue
> >>> > but when the files are copied from VB script then only issue occurs.
> Also
> >>> > the issue does not occur for the first file but it occurs randomly
> after
> >>> > processing number of files.
> >>> >
> >>> > This happens on server. I tried to reproduce this issue locally but
> no
> >>> luck.
> >>> >
> >>> > Does anybody have idea about it?
> >>> >
> >>> > Thanks!
> >>>
> >>>
> >>>
> >>> --
> >>> Claus Ibsen
> >>> -----------------
> >>> http://davsclaus.com @davsclaus
> >>> Camel in Action 2: https://www.manning.com/ibsen2
> >>>
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>
Reply | Threaded
Open this post in threaded view
|

Re: Camel lock gets acquired on files but not picked sometime

Hemant Pekhale
Thanks for the replies guys.

Actually processing of files is realtime and there is one more process
other than vb script which creates files in source folder so not feasible
with done filename.
Need to try with file consumer options only.



On Mon, Aug 6, 2018 at 9:10 PM, Trilok Agarwal <[hidden email]>
wrote:

> You can make the processes mutually exclusive through setting up a
> scheduler by giving a cron expression so that your FTP or File component
> would not read files when they are being written, just in case
> if processing of your file not real time and copying of file via VB script
> is also scheduled.
>
> Otherwise readLock=changed would help, try it out-
> From camel documentation-
>
> *changed* is using file length/modification timestamp to detect whether the
> file is currently being copied or not. Will at least use 1 sec. to
> determine this, so this option cannot consume files as fast as the others,
> but can be more reliable as the JDK IO API cannot always determine whether
> a file is currently being used by another process. The option
> *readLockCheckInterval* can be used to set the check frequency. This option
> is *only* avail for the FTP <http://camel.apache.org/ftp2.html> component
> from *Camel 2.8* onward. Note: from *Camel 2.10.1* onward the FTP
> <http://camel.apache.org/ftp2.html> option *fastExistsCheck* can be
> enabled
> to speedup this *readLock* strategy, if the FTP server support the LIST
> operation with a full file name (some servers may not)
>
> On Mon, Aug 6, 2018 at 7:26 PM, Claus Ibsen <[hidden email]> wrote:
>
> > Also you can use "done" files, eg in the VB script you write a 2nd
> > empty file with a special name, that tells Camel that the other file
> > is done being copied and ready.
> > There is some options on Camel file endpoint you can configure for that
> > also.
> >
> > On Mon, Aug 6, 2018 at 3:54 PM, Claus Ibsen <[hidden email]>
> wrote:
> > > Hi
> > >
> > > You can configure Camel to only pickup files with a certain file
> > > pattern, or to exclude names with temporary names, etc.
> > > There is a number of different filter options, see its doc.
> > >
> > > On Mon, Aug 6, 2018 at 3:52 PM, Hemant Pekhale <
> [hidden email]>
> > wrote:
> > >> Hi Claus,
> > >>
> > >> Thanks for the reply.
> > >>
> > >> Yes, I am able to modify the script. But if we do copy the file with
> > >> temporary name and then rename, I doubt that there may be lock
> acquired
> > on
> > >> temporary file name already. because my route has no waiting
> parameter.
> > So
> > >> I think it might not be helpful for our case. Please correct me if I
> am
> > >> wrong.
> > >>
> > >> Regarding the file lock strategies, I think  readLock think
> > >> readLockCheckInterval think can be tried. Do you have any other
> > suggestion?
> > >>
> > >> FYI : Whatever I can try I need to try on Production server so just
> > wanted
> > >> to tackle this with minimum and required trials.
> > >>
> > >> Thanks!
> > >>
> > >>
> > >> On Mon, Aug 6, 2018 at 7:10 PM, Claus Ibsen <[hidden email]>
> > wrote:
> > >>
> > >>> Hi
> > >>>
> > >>> Are you able to modify the VB script so it copies the file using a
> > >>> temporary name and then rename the file when the copy is done.
> > >>>
> > >>> It sounds a bit like the problem is that the VB script is currently
> > >>> still copying the file, while Camel gets a lock on the file, and
> > >>> attempt to process it.
> > >>> File locks is not as easy and it depends a bit what file system, OS,
> > >>> and how big the files are and other things.
> > >>>
> > >>> A good strategy is to copy the files using temporary names or in
> > >>> another folder, and then do a file rename when the copy is over, that
> > >>> latter then is an atomic file operation which should not cause file
> > >>> locking issues.
> > >>>
> > >>> You can also try to use some of the other Camel file lock strategies
> > >>> instead of marker files.
> > >>> You can see more in the file documentation about these.
> > >>>
> > >>>
> > >>> On Mon, Aug 6, 2018 at 3:19 PM, Hemant Pekhale <
> > [hidden email]>
> > >>> wrote:
> > >>> > Hi All,
> > >>> >
> > >>> > I have a route which picks files from a folder and do some
> > processing on
> > >>> it
> > >>> > and store it in other folder. I have used parameter "delete=true"
> for
> > >>> > deleting file from source folder.
> > >>> >
> > >>> > Sometime it happens that files are stuck in the source folder and
> > camel
> > >>> > lock files are also created for all the files but files are neither
> > >>> process
> > >>> > nor gets deleted.
> > >>> >
> > >>> > There is a VB script which copied files in the source folder. When
> I
> > >>> > manually copied the files in the source folder then it never gives
> > issue
> > >>> > but when the files are copied from VB script then only issue
> occurs.
> > Also
> > >>> > the issue does not occur for the first file but it occurs randomly
> > after
> > >>> > processing number of files.
> > >>> >
> > >>> > This happens on server. I tried to reproduce this issue locally but
> > no
> > >>> luck.
> > >>> >
> > >>> > Does anybody have idea about it?
> > >>> >
> > >>> > Thanks!
> > >>>
> > >>>
> > >>>
> > >>> --
> > >>> Claus Ibsen
> > >>> -----------------
> > >>> http://davsclaus.com @davsclaus
> > >>> Camel in Action 2: https://www.manning.com/ibsen2
> > >>>
> > >
> > >
> > >
> > > --
> > > Claus Ibsen
> > > -----------------
> > > http://davsclaus.com @davsclaus
> > > Camel in Action 2: https://www.manning.com/ibsen2
> >
> >
> >
> > --
> > Claus Ibsen
> > -----------------
> > http://davsclaus.com @davsclaus
> > Camel in Action 2: https://www.manning.com/ibsen2
> >
>
Regards,
Hemant Pekhale