Camel marker files ".camelLock" not deleted

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Camel marker files ".camelLock" not deleted

Stephan Burkard
Hi Camel riders

I have a quite simple polling file consumer that runs fine on 2 instances.

from(fileEndpointUri)
.routeId(routeId)
.log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
.setBody(method(bomProcessor, "removeBomIfPresent"))
.to(jmsEndpointUri);


The fileUri options are:
- initialDelay=[random-range]
- delay=[configurable]
- antFilterCaseSensitive=false
- antInclude=*.xml
- preMove=processing
- delete=true
- moveFailed=../error

I generate a random start delay to avoid both consumers running at the same
time. However sometimes they overlap and want to process the same file.

In this case I see normal processing in the logfile of one instance and an
error in the log of the other instance. So it works perfectly. With one
small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
That makes the operation guys nervous.

Can I do something so that these marker files are deleted? Is this kind of
a misconfiguration of my file endpoint?

Thanks a lot
Stephan
Reply | Threaded
Open this post in threaded view
|

Re: Camel marker files ".camelLock" not deleted

Zoran Regvart-2
Hi Stephan,
try setting `readLockDeleteOrphanLockFiles` on the file endpoint. Also
you might want to use an idempotent repository and a shared resource
to isolate the consumers -- for instance a database table.

zoran

On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <[hidden email]> wrote:

> Hi Camel riders
>
> I have a quite simple polling file consumer that runs fine on 2 instances.
>
> from(fileEndpointUri)
> .routeId(routeId)
> .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> .setBody(method(bomProcessor, "removeBomIfPresent"))
> .to(jmsEndpointUri);
>
>
> The fileUri options are:
> - initialDelay=[random-range]
> - delay=[configurable]
> - antFilterCaseSensitive=false
> - antInclude=*.xml
> - preMove=processing
> - delete=true
> - moveFailed=../error
>
> I generate a random start delay to avoid both consumers running at the same
> time. However sometimes they overlap and want to process the same file.
>
> In this case I see normal processing in the logfile of one instance and an
> error in the log of the other instance. So it works perfectly. With one
> small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
> That makes the operation guys nervous.
>
> Can I do something so that these marker files are deleted? Is this kind of
> a misconfiguration of my file endpoint?
>
> Thanks a lot
> Stephan



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

Re: Camel marker files ".camelLock" not deleted

Claus Ibsen-2
In reply to this post by Stephan Burkard
Hi

You can use shuffle=true to randomize the file order to spread the
chance of clash.

But when having competing consumers on the same file share you can end
up with clashes.
You can use a distributed coordinator which is covered in the 2nd ed
of the Camel book in the transactional chapter. And as well in the
upcoming chapter on clustering.



On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <[hidden email]> wrote:

> Hi Camel riders
>
> I have a quite simple polling file consumer that runs fine on 2 instances.
>
> from(fileEndpointUri)
> .routeId(routeId)
> .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> .setBody(method(bomProcessor, "removeBomIfPresent"))
> .to(jmsEndpointUri);
>
>
> The fileUri options are:
> - initialDelay=[random-range]
> - delay=[configurable]
> - antFilterCaseSensitive=false
> - antInclude=*.xml
> - preMove=processing
> - delete=true
> - moveFailed=../error
>
> I generate a random start delay to avoid both consumers running at the same
> time. However sometimes they overlap and want to process the same file.
>
> In this case I see normal processing in the logfile of one instance and an
> error in the log of the other instance. So it works perfectly. With one
> small flaw: the marker files of Camel (whatever.camelLock) are not deleted.
> That makes the operation guys nervous.
>
> Can I do something so that these marker files are deleted? Is this kind of
> a misconfiguration of my file endpoint?
>
> Thanks a lot
> Stephan



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Camel marker files ".camelLock" not deleted

Stephan Burkard
Hi Zoran and Claus

Thanks for the suggestions.

readLockDeleteOrphanLockFiles: I am on Camel 2.12, so this is currently no
option.
shuffle: dito

I will look up the coordinator in the Camel book. This is a good reason to
download the most current MEAP version :-)

Regards
Stephan


Claus Ibsen <[hidden email]> schrieb am Mo., 13. März 2017 um
17:02 Uhr:

> Hi
>
> You can use shuffle=true to randomize the file order to spread the
> chance of clash.
>
> But when having competing consumers on the same file share you can end
> up with clashes.
> You can use a distributed coordinator which is covered in the 2nd ed
> of the Camel book in the transactional chapter. And as well in the
> upcoming chapter on clustering.
>
>
>
> On Mon, Mar 13, 2017 at 3:47 PM, Stephan Burkard <[hidden email]>
> wrote:
> > Hi Camel riders
> >
> > I have a quite simple polling file consumer that runs fine on 2
> instances.
> >
> > from(fileEndpointUri)
> > .routeId(routeId)
> > .log("[FILEREADER " + routeId + "], file: ${in.header.CamelFileName}")
> > .setBody(method(bomProcessor, "removeBomIfPresent"))
> > .to(jmsEndpointUri);
> >
> >
> > The fileUri options are:
> > - initialDelay=[random-range]
> > - delay=[configurable]
> > - antFilterCaseSensitive=false
> > - antInclude=*.xml
> > - preMove=processing
> > - delete=true
> > - moveFailed=../error
> >
> > I generate a random start delay to avoid both consumers running at the
> same
> > time. However sometimes they overlap and want to process the same file.
> >
> > In this case I see normal processing in the logfile of one instance and
> an
> > error in the log of the other instance. So it works perfectly. With one
> > small flaw: the marker files of Camel (whatever.camelLock) are not
> deleted.
> > That makes the operation guys nervous.
> >
> > Can I do something so that these marker files are deleted? Is this kind
> of
> > a misconfiguration of my file endpoint?
> >
> > Thanks a lot
> > Stephan
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>