"Can not delete file" when combining file poller and JMS

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

"Can not delete file" when combining file poller and JMS

marek.vaclavik
Hello,

I have encountered an issue when combining file poller and JMS queue in one route. If the poller is set to "noop=no" and tries to either delete or move the files, it fails every now and then (say, every 5th to 10th file). For instance, with "noop=no&delete=yes" on a Windows machine the exception thrown is: "2009-03-27T17:13:59,781 [Thread: 1 org.apache.camel.component.file.FileComponent@1af5350] WARN  - component.file.FileConsumer | Error committing file strategy: org.apache.camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6
java.io.IOException: Can not delete file: .\data\pxsa\transfer\in\file.txt"

Apparently, the issue has been discussed in in various Camel forums:
- http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html
- http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html
- http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@mail.gmail.com%3E
- http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030

Unfortunately, none of the solutions suggested there (using delete=true, converting body to string) worked for us 100%. Adding a delay reduced the occurence of the exception, though. Setting "consumer.exclusiveReadLock=false" is no option for us, as we want to prevent incomplete files from being read.

Removing the JMS endpoint from the route in our application prevents the error, which indicates that the cause is connected to JMS. This is also our workaround at the moment.

I wonder whether
a) the problem is familiar to somebody?
b) there is a "trick", an endpoint setting or an addition to the route, that reliably eliminates the problem?

Any help is greatly appreciated.

Kind Regards
Marek
Reply | Threaded
Open this post in threaded view
|

Re: "Can not delete file" when combining file poller and JMS

Bruno Borges
Have you tried to put a direct:endpoint in the middle, between file:* and  
jms:* ?

Before routing to direct:, convert body to String. Then you route  
from("direct:endpoint").to("jms:queue")

Maybe this works.

Cheers,
Bruno Borges

On Apr 8, 2009 11:39am, Marek2009 <[hidden email]> wrote:


> Hello,



> I have encountered an issue when combining file poller and JMS queue in  
> one

> route. If the poller is set to "noop=no" and tries to either delete or  
> move

> the files, it fails every now and then (say, every 5th to 10th file). For

> instance, with "noop=no&delete=yes" on a Windows machine the exception

> thrown is: "2009-03-27T17:13:59,781 [Thread: 1

> org.apache.camel.component.file.FileComponent@1af5350] WARN -

> component.file.FileConsumer | Error committing file strategy:

> org.apache.camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6

> java.io.IOException: Can not delete  
> file: .\data\pxsa\transfer\in\file.txt"



> Apparently, the issue has been discussed in in various Camel forums:

> - http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html

> - http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html

> -

> http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@...%3E

> - http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030



> Unfortunately, none of the solutions suggested there (using delete=true,

> converting body to string) worked for us 100%. Adding a delay reduced the

> occurence of the exception, though. Setting

> "consumer.exclusiveReadLock=false" is no option for us, as we want to

> prevent incomplete files from being read.



> Removing the JMS endpoint from the route in our application prevents the

> error, which indicates that the cause is connected to JMS. This is also  
> our

> workaround at the moment.



> I wonder whether

> a) the problem is familiar to somebody?

> b) there is a "trick", an endpoint setting or an addition to the route,  
> that

> reliably eliminates the problem?



> Any help is greatly appreciated.



> Kind Regards

> Marek



> --

> View this message in context:  
> http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p22951942.html

> Sent from the Camel - Users mailing list archive at Nabble.com.



Reply | Threaded
Open this post in threaded view
|

Re: "Can not delete file" when combining file poller and JMS

Adrian Trenaman-2
Hi Bruno, Marek,

Yeah, I encountered someting similar with Camel 1.5.4.0-fuse on  
Windows: if you convertBodyTo(String.class) after the from("file:...")  
then it should be ok.

/Ade
On 8 Apr 2009, at 15:50, [hidden email] wrote:

> Have you tried to put a direct:endpoint in the middle, between  
> file:* and jms:* ?
>
> Before routing to direct:, convert body to String. Then you route  
> from("direct:endpoint").to("jms:queue")
>
> Maybe this works.
>
> Cheers,
> Bruno Borges
>
> On Apr 8, 2009 11:39am, Marek2009 <[hidden email]> wrote:
>
>
>> Hello,
>
>
>
>> I have encountered an issue when combining file poller and JMS  
>> queue in one
>
>> route. If the poller is set to "noop=no" and tries to either delete  
>> or move
>
>> the files, it fails every now and then (say, every 5th to 10th  
>> file). For
>
>> instance, with "noop=no&delete=yes" on a Windows machine the  
>> exception
>
>> thrown is: "2009-03-27T17:13:59,781 [Thread: 1
>
>> org.apache.camel.component.file.FileComponent@1af5350] WARN -
>
>> component.file.FileConsumer | Error committing file strategy:
>
>> org
>> .apache
>> .camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6
>
>> java.io.IOException: Can not delete file: .\data\pxsa\transfer\in
>> \file.txt"
>
>
>
>> Apparently, the issue has been discussed in in various Camel forums:
>
>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html
>
>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html
>
>> -
>
>> http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@...%3E
>
>> - http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030
>
>
>
>> Unfortunately, none of the solutions suggested there (using  
>> delete=true,
>
>> converting body to string) worked for us 100%. Adding a delay  
>> reduced the
>
>> occurence of the exception, though. Setting
>
>> "consumer.exclusiveReadLock=false" is no option for us, as we want to
>
>> prevent incomplete files from being read.
>
>
>
>> Removing the JMS endpoint from the route in our application  
>> prevents the
>
>> error, which indicates that the cause is connected to JMS. This is  
>> also our
>
>> workaround at the moment.
>
>
>
>> I wonder whether
>
>> a) the problem is familiar to somebody?
>
>> b) there is a "trick", an endpoint setting or an addition to the  
>> route, that
>
>> reliably eliminates the problem?
>
>
>
>> Any help is greatly appreciated.
>
>
>
>> Kind Regards
>
>> Marek
>
>
>
>> --
>
>> View this message in context: http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p22951942.html
>
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

---
Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
Progress Software Corp
Shelbourne Road, Dublin 4, Ireland
---
+353-1-637-2659 (Office)
+353-1-637-2882 (Fax)
+353-86-6051026 (Mobile)
  adrian.trenaman (Skype)
----
Blog: http://trenaman.blogspot.com








Reply | Threaded
Open this post in threaded view
|

Re: "Can not delete file" when combining file poller and JMS

Claus Ibsen-2
Hi

Yeah I have tracked it down to a potential buffer stream not being
closed, that is used by the JMS consumer.
Converting the file content to String based beforehand should fix it
as Ade suggested.

However the bugfix is committed to 1.6.1 and 2.0 so hopefully it
should work out of the box in the future.


On Wed, Apr 8, 2009 at 4:54 PM, Adrian Trenaman <[hidden email]> wrote:

> Hi Bruno, Marek,
>
> Yeah, I encountered someting similar with Camel 1.5.4.0-fuse on Windows: if
> you convertBodyTo(String.class) after the from("file:...") then it should be
> ok.
>
> /Ade
> On 8 Apr 2009, at 15:50, [hidden email] wrote:
>
>> Have you tried to put a direct:endpoint in the middle, between file:* and
>> jms:* ?
>>
>> Before routing to direct:, convert body to String. Then you route
>> from("direct:endpoint").to("jms:queue")
>>
>> Maybe this works.
>>
>> Cheers,
>> Bruno Borges
>>
>> On Apr 8, 2009 11:39am, Marek2009 <[hidden email]> wrote:
>>
>>
>>> Hello,
>>
>>
>>
>>> I have encountered an issue when combining file poller and JMS queue in
>>> one
>>
>>> route. If the poller is set to "noop=no" and tries to either delete or
>>> move
>>
>>> the files, it fails every now and then (say, every 5th to 10th file). For
>>
>>> instance, with "noop=no&delete=yes" on a Windows machine the exception
>>
>>> thrown is: "2009-03-27T17:13:59,781 [Thread: 1
>>
>>> org.apache.camel.component.file.FileComponent@1af5350] WARN -
>>
>>> component.file.FileConsumer | Error committing file strategy:
>>
>>>
>>> org.apache.camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6
>>
>>> java.io.IOException: Can not delete file:
>>> .\data\pxsa\transfer\in\file.txt"
>>
>>
>>
>>> Apparently, the issue has been discussed in in various Camel forums:
>>
>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html
>>
>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html
>>
>>> -
>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@...%3E
>>
>>> - http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030
>>
>>
>>
>>> Unfortunately, none of the solutions suggested there (using delete=true,
>>
>>> converting body to string) worked for us 100%. Adding a delay reduced the
>>
>>> occurence of the exception, though. Setting
>>
>>> "consumer.exclusiveReadLock=false" is no option for us, as we want to
>>
>>> prevent incomplete files from being read.
>>
>>
>>
>>> Removing the JMS endpoint from the route in our application prevents the
>>
>>> error, which indicates that the cause is connected to JMS. This is also
>>> our
>>
>>> workaround at the moment.
>>
>>
>>
>>> I wonder whether
>>
>>> a) the problem is familiar to somebody?
>>
>>> b) there is a "trick", an endpoint setting or an addition to the route,
>>> that
>>
>>> reliably eliminates the problem?
>>
>>
>>
>>> Any help is greatly appreciated.
>>
>>
>>
>>> Kind Regards
>>
>>> Marek
>>
>>
>>
>>> --
>>
>>> View this message in context:
>>> http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p22951942.html
>>
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>
> ---
> Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
> Progress Software Corp
> Shelbourne Road, Dublin 4, Ireland
> ---
> +353-1-637-2659 (Office)
> +353-1-637-2882 (Fax)
> +353-86-6051026 (Mobile)
>  adrian.trenaman (Skype)
> ----
> Blog: http://trenaman.blogspot.com
>
>
>
>
>
>
>
>
>



--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Reply | Threaded
Open this post in threaded view
|

Re: "Can not delete file" when combining file poller and JMS

marek.vaclavik
Hello Klaus, Adrian, Bruno,

although the workaround you suggest is trivial, it took us a while to implement and test it. :-)

I would like to confirm that inserting a "convertBodyTo(String.class)" worked for us (Camel 1.6.0).
Since the modification we have not encountered the file access issue in any of our tests, including load tests with larger numbers of files.

We hope to be able to remove the workaround code from the route, as soon as we upgrade to Camel 1.6.1 or later. :-)

Many thanks for your help!
Kind Regards

Marek

Claus Ibsen-2 wrote
Hi

Yeah I have tracked it down to a potential buffer stream not being
closed, that is used by the JMS consumer.
Converting the file content to String based beforehand should fix it
as Ade suggested.

However the bugfix is committed to 1.6.1 and 2.0 so hopefully it
should work out of the box in the future.


On Wed, Apr 8, 2009 at 4:54 PM, Adrian Trenaman <trenaman@progress.com> wrote:
> Hi Bruno, Marek,
>
> Yeah, I encountered someting similar with Camel 1.5.4.0-fuse on Windows: if
> you convertBodyTo(String.class) after the from("file:...") then it should be
> ok.
>
> /Ade
> On 8 Apr 2009, at 15:50, bruno.borges@gmail.com wrote:
>
>> Have you tried to put a direct:endpoint in the middle, between file:* and
>> jms:* ?
>>
>> Before routing to direct:, convert body to String. Then you route
>> from("direct:endpoint").to("jms:queue")
>>
>> Maybe this works.
>>
>> Cheers,
>> Bruno Borges
>>
>> On Apr 8, 2009 11:39am, Marek2009 <Marek.Vaclavik@icw.de> wrote:
>>
>>
>>> Hello,
>>
>>
>>
>>> I have encountered an issue when combining file poller and JMS queue in
>>> one
>>
>>> route. If the poller is set to "noop=no" and tries to either delete or
>>> move
>>
>>> the files, it fails every now and then (say, every 5th to 10th file). For
>>
>>> instance, with "noop=no&delete=yes" on a Windows machine the exception
>>
>>> thrown is: "2009-03-27T17:13:59,781 [Thread: 1
>>
>>> org.apache.camel.component.file.FileComponent@1af5350] WARN -
>>
>>> component.file.FileConsumer | Error committing file strategy:
>>
>>>
>>> org.apache.camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6
>>
>>> java.io.IOException: Can not delete file:
>>> .\data\pxsa\transfer\in\file.txt"
>>
>>
>>
>>> Apparently, the issue has been discussed in in various Camel forums:
>>
>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html
>>
>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html
>>
>>> -
>>
>>>
>>> http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@mail.gmail.com%3E
>>
>>> - http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030
>>
>>
>>
>>> Unfortunately, none of the solutions suggested there (using delete=true,
>>
>>> converting body to string) worked for us 100%. Adding a delay reduced the
>>
>>> occurence of the exception, though. Setting
>>
>>> "consumer.exclusiveReadLock=false" is no option for us, as we want to
>>
>>> prevent incomplete files from being read.
>>
>>
>>
>>> Removing the JMS endpoint from the route in our application prevents the
>>
>>> error, which indicates that the cause is connected to JMS. This is also
>>> our
>>
>>> workaround at the moment.
>>
>>
>>
>>> I wonder whether
>>
>>> a) the problem is familiar to somebody?
>>
>>> b) there is a "trick", an endpoint setting or an addition to the route,
>>> that
>>
>>> reliably eliminates the problem?
>>
>>
>>
>>> Any help is greatly appreciated.
>>
>>
>>
>>> Kind Regards
>>
>>> Marek
>>
>>
>>
>>> --
>>
>>> View this message in context:
>>> http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p22951942.html
>>
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>
> ---
> Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
> Progress Software Corp
> Shelbourne Road, Dublin 4, Ireland
> ---
> +353-1-637-2659 (Office)
> +353-1-637-2882 (Fax)
> +353-86-6051026 (Mobile)
>  adrian.trenaman (Skype)
> ----
> Blog: http://trenaman.blogspot.com
>
>
>
>
>
>
>
>
>



--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Apache Camel Reference Card:
http://refcardz.dzone.com/refcardz/enterprise-integration
Reply | Threaded
Open this post in threaded view
|

Re: "Can not delete file" when combining file poller and JMS

Claus Ibsen-2
On Fri, Jun 19, 2009 at 5:49 PM, Marek2009 <[hidden email]> wrote:

>
> Hello Klaus, Adrian, Bruno,
>
> although the workaround you suggest is trivial, it took us a while to
> implement and test it. :-)
>
> I would like to confirm that inserting a "convertBodyTo(String.class)"
> worked for us (Camel 1.6.0).
> Since the modification we have not encountered the file access issue in any
> of our tests, including load tests with larger numbers of files.
>
> We hope to be able to remove the workaround code from the route, as soon as
> we upgrade to Camel 1.6.1 or later. :-)
>

Hi Marek

Thanks for sharing this. Nice to know the workaround worked.
Fingers crossed that its fixed in 1.6.1 with that missing close().




>
> Many thanks for your help!
> Kind Regards
>
> Marek
>
>
> Claus Ibsen-2 wrote:
> >
> > Hi
> >
> > Yeah I have tracked it down to a potential buffer stream not being
> > closed, that is used by the JMS consumer.
> > Converting the file content to String based beforehand should fix it
> > as Ade suggested.
> >
> > However the bugfix is committed to 1.6.1 and 2.0 so hopefully it
> > should work out of the box in the future.
> >
> >
> > On Wed, Apr 8, 2009 at 4:54 PM, Adrian Trenaman <[hidden email]>
> > wrote:
> >> Hi Bruno, Marek,
> >>
> >> Yeah, I encountered someting similar with Camel 1.5.4.0-fuse on Windows:
> >> if
> >> you convertBodyTo(String.class) after the from("file:...") then it
> should
> >> be
> >> ok.
> >>
> >> /Ade
> >> On 8 Apr 2009, at 15:50, [hidden email] wrote:
> >>
> >>> Have you tried to put a direct:endpoint in the middle, between file:*
> >>> and
> >>> jms:* ?
> >>>
> >>> Before routing to direct:, convert body to String. Then you route
> >>> from("direct:endpoint").to("jms:queue")
> >>>
> >>> Maybe this works.
> >>>
> >>> Cheers,
> >>> Bruno Borges
> >>>
> >>> On Apr 8, 2009 11:39am, Marek2009 <[hidden email]> wrote:
> >>>
> >>>
> >>>> Hello,
> >>>
> >>>
> >>>
> >>>> I have encountered an issue when combining file poller and JMS queue
> in
> >>>> one
> >>>
> >>>> route. If the poller is set to "noop=no" and tries to either delete or
> >>>> move
> >>>
> >>>> the files, it fails every now and then (say, every 5th to 10th file).
> >>>> For
> >>>
> >>>> instance, with "noop=no&delete=yes" on a Windows machine the exception
> >>>
> >>>> thrown is: "2009-03-27T17:13:59,781 [Thread: 1
> >>>
> >>>> org.apache.camel.component.file.FileComponent@1af5350] WARN -
> >>>
> >>>> component.file.FileConsumer | Error committing file strategy:
> >>>
> >>>>
> >>>>
> org.apache.camel.component.file.strategy.DeleteFileProcessStrategy@10c81a6
> >>>
> >>>> java.io.IOException: Can not delete file:
> >>>> .\data\pxsa\transfer\in\file.txt"
> >>>
> >>>
> >>>
> >>>> Apparently, the issue has been discussed in in various Camel forums:
> >>>
> >>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00119.html
> >>>
> >>>> - http://osdir.com/ml/users-camel-apache/2009-02/msg00123.html
> >>>
> >>>> -
> >>>
> >>>>
> >>>>
> http://mail-archives.apache.org/mod_mbox/camel-users/200902.mbox/%3C5380c69c0902130209l679c3ab0j641036484ee14875@...%3E
> >>>
> >>>> - http://cwiki.apache.org/SM/discussion-forums.html#nabble-p21504030
> >>>
> >>>
> >>>
> >>>> Unfortunately, none of the solutions suggested there (using
> >>>> delete=true,
> >>>
> >>>> converting body to string) worked for us 100%. Adding a delay reduced
> >>>> the
> >>>
> >>>> occurence of the exception, though. Setting
> >>>
> >>>> "consumer.exclusiveReadLock=false" is no option for us, as we want to
> >>>
> >>>> prevent incomplete files from being read.
> >>>
> >>>
> >>>
> >>>> Removing the JMS endpoint from the route in our application prevents
> >>>> the
> >>>
> >>>> error, which indicates that the cause is connected to JMS. This is
> also
> >>>> our
> >>>
> >>>> workaround at the moment.
> >>>
> >>>
> >>>
> >>>> I wonder whether
> >>>
> >>>> a) the problem is familiar to somebody?
> >>>
> >>>> b) there is a "trick", an endpoint setting or an addition to the
> route,
> >>>> that
> >>>
> >>>> reliably eliminates the problem?
> >>>
> >>>
> >>>
> >>>> Any help is greatly appreciated.
> >>>
> >>>
> >>>
> >>>> Kind Regards
> >>>
> >>>> Marek
> >>>
> >>>
> >>>
> >>>> --
> >>>
> >>>> View this message in context:
> >>>>
> http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p22951942.html
> >>>
> >>>> Sent from the Camel - Users mailing list archive at Nabble.com.
> >>>
> >>>
> >>>
> >>
> >> ---
> >> Adrian Trenaman, Consultant Fellow, PS - Opensource Center of Competence
> >> Progress Software Corp
> >> Shelbourne Road, Dublin 4, Ireland
> >> ---
> >> +353-1-637-2659 (Office)
> >> +353-1-637-2882 (Fax)
> >> +353-86-6051026 (Mobile)
> >>  adrian.trenaman (Skype)
> >> ----
> >> Blog: http://trenaman.blogspot.com
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >>
> >
> >
> >
> > --
> > Claus Ibsen
> > Apache Camel Committer
> >
> > Open Source Integration: http://fusesource.com
> > Blog: http://davsclaus.blogspot.com/
> > Twitter: http://twitter.com/davsclaus
> > Apache Camel Reference Card:
> > http://refcardz.dzone.com/refcardz/enterprise-integration
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/%22Can-not-delete-file%22-when-combining-file-poller-and-JMS-tp22951942p24114062.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--
Claus Ibsen
Apache Camel Committer

Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus