[jira] Created: (CAMEL-1099) File consumer should keep track of processed files

File consumer should keep track of processed files
--------------------------------------------------

                 Key: CAMEL-1099
                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
             Project: Apache Camel
          Issue Type: Improvement
          Components: camel-core, camel-ftp, camel-spring
            Reporter: Jonathan Anstey
            Assignee: Jonathan Anstey
             Fix For: 2.0.0


See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html

Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
- an in-memory store for testing purposes
- a file-based implementation for basic production environments
- a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories


--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47698#action_47698 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

I got a bit deep into the code today in the FileConsumer and have added options for *idempotent* so you can keep track of already consumed files.

See my new post on the nabble forum with this idea.

Jon do you mind review the commited source code on trunk. Sorry for invading this ticket, but the re factor of the file consumer needed to be done in one go as to have most unit test still passing.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47698#action_47698 ]

davsclaus edited comment on CAMEL-1099 at 11/29/08 9:13 AM:
--------------------------------------------------------------

I got a bit deep into the code today in the FileConsumer and have added options for *idempotent* so you can keep track of already consumed files.

See my new post on the nabble forum with this idea.

Jon do you mind review the commited source code on trunk. Sorry for invading this ticket, but the re factor of the file consumer needed to be done in one go as to have most unit test still passing. The commits will be on CAMEL-1112

      was (Author: davsclaus):
    I got a bit deep into the code today in the FileConsumer and have added options for *idempotent* so you can keep track of already consumed files.

See my new post on the nabble forum with this idea.

Jon do you mind review the commited source code on trunk. Sorry for invading this ticket, but the re factor of the file consumer needed to be done in one go as to have most unit test still passing.
 

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47700#action_47700 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

To review:
- I have set the default cache size of the memory to 1000, is this a good default value?
- FileConsumer code
- file.getName is that a sufficient key for idempotent check (should we use absolute file name instead?)
- Is the header keys for index/total a good key name?
- Should idempotent be enabled by default?

Notice for unit tests that want to consume files but not delete: noop=true&idempotent=true will be as before, only consuming the file ones and not delete the source files.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47739#action_47739 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

MessageIdRepository should be renamed and moved to org.apache.camel.spi
MessageIdRepository -> IdempotentRepository

See nabble post with James suggestion for this.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47740#action_47740 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

James had a good idea about the interface methods

How about following the Set naming convention?

boolean add(element) // Returns true if this set did not already
contain the specified element.

boolean contains(element) // for peek

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47747#action_47747 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

BTW: I am currently working on moving the idempotent repository to the spi package.

Should we support generic, like the Set contract?

Should it be:
public interface IdempotentRepository<E>
Or without the <E> and using Object as the key type?

I currently have the generics version

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47750#action_47750 ]

Jonathan Anstey commented on CAMEL-1099:
----------------------------------------

I know we've been removing generics all over the place in Camel lately... I like it in this case though. +1 to your current code.

Claus, you've pretty much completed this task so I'm assigning it to you so you get the credit :)

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Jonathan Anstey
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jonathan Anstey reassigned CAMEL-1099:
--------------------------------------

    Assignee: Claus Ibsen  (was: Jonathan Anstey)

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47752#action_47752 ]

Hadrian Zbarcea commented on CAMEL-1099:
----------------------------------------

+1

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47753#action_47753 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

Jon, let's have joint credit, we are a team after all ;)

I am wondering if we / I should dig deeper to do something with the MemoryIdempotentRepository as it states that you should be careful with the cache Map to avoid memory leeks.

I was wondering if we could get the Service interface with start/stop to eg. clear the cache on stop. But I guess the callback of Service is not invoked on the idempotent node. Well something to think about.

Any thoughts?

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47755#action_47755 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

Maybe noop=true should also make idempotent=true. It makes sense *not* to process the same file over and over again.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

     [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on CAMEL-1099 started by Claus Ibsen.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47756#action_47756 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

D:\project\camel>svn commit --message "CAMEL-1099: Introduced IdempotentRepository in the spi package."
Sending        camel-core\src\main\java\org\apache\camel\component\file\FileComponent.java
Sending        camel-core\src\main\java\org\apache\camel\component\file\FileConsumer.java
Sending        camel-core\src\main\java\org\apache\camel\component\file\FileEndpoint.java
Sending        camel-core\src\main\java\org\apache\camel\model\IdempotentConsumerType.java
Sending        camel-core\src\main\java\org\apache\camel\model\ProcessorType.java
Sending        camel-core\src\main\java\org\apache\camel\processor\idempotent\IdempotentConsumer.java
Adding         camel-core\src\main\java\org\apache\camel\processor\idempotent\MemoryIdempotentRepository.java
Deleting       camel-core\src\main\java\org\apache\camel\processor\idempotent\MemoryMessageIdRepository.java
Deleting       camel-core\src\main\java\org\apache\camel\processor\idempotent\MessageIdRepository.java
Adding         camel-core\src\main\java\org\apache\camel\spi\IdempotentRepository.java
Sending        camel-core\src\test\java\org\apache\camel\builder\RouteBuilderTest.java
Sending        camel-core\src\test\java\org\apache\camel\component\file\FileConsumerIdempotentRefTest.java
Sending        camel-core\src\test\java\org\apache\camel\processor\IdempotentConsumerTest.java
Sending        components\camel-jpa\src\main\java\org\apache\camel\processor\idempotent\jpa\JpaMessageIdRepository.java
Sending        components\camel-spring\src\test\java\org\apache\camel\component\test\TestEndpointTest.java
Sending        components\camel-spring\src\test\resources\org\apache\camel\component\test\TestEndpointTest-context.xml
Transmitting file data ..............
Committed revision 722088.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47757#action_47757 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

- Refactor idempotent interface to spi package *DONE*
- Wiki sample how to use the Jpa store provided by camel-jpa
- Add a file based repository
- Wiki sample how to use the File store

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47758#action_47758 ]

Jonathan Anstey commented on CAMEL-1099:
----------------------------------------

Claus, I wouldn't spend too much time on the MemoryIdempotentRepository, I don't think many users would want to use this in a production scenario (the file and jpa options are much more attractive). The best we can do in this case is recommend the LRU cache option so the memory doesn't get too out of hand I guess.

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47775#action_47775 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

Jon, others?

Any good ideas of the file based repository?

I am wondering if we should
- have the LRUCache as a 1st level cache, and have write through to the file store when adding, using append
- load the filestore on startup
- configurable filename and folder
- use camel file component to read/write (configuration could then be a file endpoint = "file://data/idempotentrepository.txt?append=true&autoCreate=true")

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47780#action_47780 ]

Jonathan Anstey commented on CAMEL-1099:
----------------------------------------

So when there is a miss on the LRUCache, the file repository would be queried (and this is the only case that it would queried)? I'm just wondering what kind of format we want for this kinda file... having it appended to with every add seems to imply that it would be a big list that we'd have to scroll through (very slow). Maybe I'm missing an obvious solution, it is early here :)

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47781#action_47781 ]

Jonathan Anstey commented on CAMEL-1099:
----------------------------------------

Will think about this some more when I get into the office. I bet there are some very cool file based index things in KahaDB though http://svn.apache.org/repos/asf/activemq/trunk/kahadb/

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.

    [ https://issues.apache.org/activemq/browse/CAMEL-1099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=47782#action_47782 ]

Claus Ibsen commented on CAMEL-1099:
------------------------------------

Well the file list will be loaded at startup and then only used for write when there is a cache hit, so there would not be a performance hit at runtime.

I think it's overkill to use some kind of 3rd party lib

> File consumer should keep track of processed files
> --------------------------------------------------
>
>                 Key: CAMEL-1099
>                 URL: https://issues.apache.org/activemq/browse/CAMEL-1099
>             Project: Apache Camel
>          Issue Type: Improvement
>          Components: camel-core, camel-ftp, camel-spring
>            Reporter: Jonathan Anstey
>            Assignee: Claus Ibsen
>             Fix For: 2.0.0
>
>
> See thread: http://www.nabble.com/Deprecation-of-file-consumer-timestamp-td20566034s22882.html
> Will be following Gert's suggestion of multiple implementations for the storage mechanism so many use cases can be covered:
> - an in-memory store for testing purposes
> - a file-based implementation for basic production environments
> - a database- or ldap-backed implementation for clustered environments, where a file can arrive through multiple directories

--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.