Camel › Camel - Users

File2 component not polling file for changes?

Classic

List

Threaded

5 messages Options

anydoby

File2 component not polling file for changes?

Hello. Perhaps I am not understanding the intent of file polling mechanism, but anyway.

What I need is to check whether the file was changed in any way after I have last read from it. The criteria is lastModified of the file was changed.

I expected that the file2 component was capable of tracking last modified of the file. Is that true? If not, is there a way in Camel to do this without writing your own component? Seems like it's a very common task what I am asking for.

Remove Ads

Claus Ibsen-2

Re: File2 component not polling file for changes?

Hi

The file component generally pickup files in a folder. Its not meant
to pickup the same file again, just because the timestamp changed.
This is really usually a bad way of transferring files. For example
what if the server crashes etc. Then you need to persist which files
you last processed and their modification timestamp.

Hence a strategy to delete/move processed files is much easier and better.

You can implement a custom idempotentRepository to tell Camel
- if the file has been processed before: return false in the contains
method when the file has been changed since last
- The API passes in the absolute file name, so you gotta use the
java.io.File to access the modification timestamp
- And then keep a Map of modification timestamp for last processed.

On Mon, Jul 19, 2010 at 8:53 AM, anydoby <[hidden email]> wrote:

>
> Hello. Perhaps I am not understanding the intent of file polling mechanism,
> but anyway.
>
> What I need is to check whether the file was changed in any way after I have
> last read from it. The criteria is lastModified of the file was changed.
>
> I expected that the file2 component was capable of tracking last modified of
> the file. Is that true? If not, is there a way in Camel to do this without
> writing your own component? Seems like it's a very common task what I am
> asking for.
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-component-not-polling-file-for-changes-tp1438631p1438631.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus

anydoby

Re: File2 component not polling file for changes?

Hi Claus,

Thank you for the explanation. I already realized how to do this using an idempotent thingie, however the IdempotentRepository.add is only provided with a file name, not an absolute file path, as you said (GenericFileConsumer.isValidFile). At least in Camel 2.2, so using the idempotentRepository is no-go.

        } else if (endpoint.isIdempotent() && endpoint.getIdempotentRepository().contains(file.getFileName())) {
            // only use the filename as the key as the file could be moved into a done folder
            if (log.isTraceEnabled()) {
                log.trace("This consumer is idempotent and the file has been consumed before. Will skip this file: " + file);
            }
            return false;
        }

Notice the file.getFileName()? A bug?

What I assume I should do is something like this:


                from("file:test?noop=true&fileName=test.txt&delay=500").idempotentConsumer(new Expression()
                {

                    public <T> T evaluate(final Exchange exchange, final Class<T> type)
                    {
                        final File body = exchange.getIn().getBody(File.class);
                        final String path = body.getAbsolutePath();
                        return (T) path;
                    }
                }, new FileLastModifiedRepository()).process(processor());

Claus Ibsen-2

Re: File2 component not polling file for changes?

Hi

Yeah in 2.4 its the absolute file name. It may also have been fixed in 2.3.
I suggest to upgrade.

Or if you consume from the same path then in 2.2 you know the starting
path and can then compute the absolute path yourself.

On Mon, Jul 19, 2010 at 11:42 AM, anydoby <[hidden email]> wrote:

>
> Hi Claus,
>
> Thank you for the explanation. I already realized how to do this using an
> idempotent thingie, however the IdempotentRepository.add is only provided
> with a file name, not an absolute file path, as you said
> (GenericFileConsumer.isValidFile). At least in Camel 2.2, so using the
> idempotentRepository is no-go.
>
>
> } else if (endpoint.isIdempotent() &&
> endpoint.getIdempotentRepository().contains(file.getFileName())) {
> // only use the filename as the key as the file could be moved
> into a done folder
> if (log.isTraceEnabled()) {
> log.trace("This consumer is idempotent and the file has been
> consumed before. Will skip this file: " + file);
> }
> return false;
> }
>
>
> Notice the file.getFileName()? A bug?
>
> What I assume I should do is something like this:
>
>
>
>
> from("file:test?noop=true&fileName=test.txt&delay=500").idempotentConsumer(new
> Expression()
> {
>
> public <T> T evaluate(final Exchange exchange, final
> Class<T> type)
> {
> final File body =
> exchange.getIn().getBody(File.class);
> final String path = body.getAbsolutePath();
> return (T) path;
> }
> }, new FileLastModifiedRepository()).process(processor());
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/File2-component-not-polling-file-for-changes-tp1438631p1438707.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>

anydoby

Re: File2 component not polling file for changes?

This post was updated on .

Thank you for the options :)

Ok, I'll consider an upgrade.

Anyway thank you again for timely replies, good luck.

PS. I published a little note on how to implement tracking changes in files using Camel, as it appeared to be not toooo simple. Probably someone will be interested in it: http://anydoby.com/jblog/en/java/1931