Identifying and processing changed file when noop=true

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Identifying and processing changed file when noop=true

vijayp
I have added following implementation of IdempotentRepository to process a specific file from directory  when the file is modified. This is using Camel 2.x, Is there an alternative approach ?

<from uri="file:src/data?noop=true&amp;idempotentRepository=#fileChanged&amp;delay=300000&amp;fileName=myfile.txt"/>

<bean id="fileChanged" class="mypkg.FileChangedRepository">
        <property name="fileDir" value="src/data" />
</bean>

public class FileChangedRepository implements IdempotentRepository<String>{

        private String fileDir;
        private long lastModified =0;
       
        @Override
        public boolean add(String arg0) {
                return false;
        }

        @Override
        public boolean confirm(String arg0) {
                return true;
        }

        @Override
        public boolean contains(String arg0) {
                synchronized(this) {
  File file = new File(fileDir + File.separator + arg0);

                if (file.lastModified() > lastModified) {
                        lastModified = file.lastModified();
                        return false;
                }
                return true;
               }
        }

        @Override
        public boolean remove(String arg0) {
                return false;
        }

        public void setFileDir(String fileDir) {
                this.fileDir = fileDir;
        }

        public String getFileDir() {
                return fileDir;
        }
}
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

Claus Ibsen-2
Hi

Have you tried the readLock=changed option?
http://camel.apache.org/file2.html


On Thu, Jan 28, 2010 at 4:01 PM, vjintegrates <[hidden email]> wrote:

>
> I have added following implementation of IdempotentRepository to process a
> specific file from directory  when the file is modified. This is using Camel
> 2.x, Is there an alternative approach ?
>
> <from
> uri="file:src/data?noop=true&amp;idempotentRepository=#fileChanged&amp;delay=300000&amp;fileName=myfile.txt"/>
>
> <bean id="fileChanged" class="mypkg.FileChangedRepository">
>        <property name="fileDir" value="src/data" />
> </bean>
>
> public class FileChangedRepository implements IdempotentRepository<String>{
>
>        private String fileDir;
>        private long lastModified =0;
>
>        @Override
>        public boolean add(String arg0) {
>                return false;
>        }
>
>        @Override
>        public boolean confirm(String arg0) {
>                return true;
>        }
>
>        @Override
>        public boolean contains(String arg0) {
>                synchronized(this) {
>                File file = new File(fileDir + File.separator + arg0);
>
>                if (file.lastModified() > lastModified) {
>                        lastModified = file.lastModified();
>                        return false;
>                }
>                return true;
>               }
>        }
>
>        @Override
>        public boolean remove(String arg0) {
>                return false;
>        }
>
>        public void setFileDir(String fileDir) {
>                this.fileDir = fileDir;
>        }
>
>        public String getFileDir() {
>                return fileDir;
>        }
> }
> --
> View this message in context: http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27357357.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

vijayp
I had tried this option it does not result in file being processed based on timestamp.

readLock=changed

I believe this attribute is used to to check if the file can be locked or not depending on last modified timestamp.

Changed file were processed automatically in Camel 1.x, once I upgraded to 2.x it stopped working, therefore I decided to provide the implementation of  IdempotentRepository


Claus Ibsen-2 wrote
Hi

Have you tried the readLock=changed option?
http://camel.apache.org/file2.html


On Thu, Jan 28, 2010 at 4:01 PM, vjintegrates <vijaypawnarkar@gmail.com> wrote:
>
> I have added following implementation of IdempotentRepository to process a
> specific file from directory  when the file is modified. This is using Camel
> 2.x, Is there an alternative approach ?
>
> <from
> uri="file:src/data?noop=true&idempotentRepository=#fileChanged&delay=300000&fileName=myfile.txt"/>
>
> <bean id="fileChanged" class="mypkg.FileChangedRepository">
>        <property name="fileDir" value="src/data" />
> </bean>
>
> public class FileChangedRepository implements IdempotentRepository<String>{
>
>        private String fileDir;
>        private long lastModified =0;
>
>        @Override
>        public boolean add(String arg0) {
>                return false;
>        }
>
>        @Override
>        public boolean confirm(String arg0) {
>                return true;
>        }
>
>        @Override
>        public boolean contains(String arg0) {
>                synchronized(this) {
>                File file = new File(fileDir + File.separator + arg0);
>
>                if (file.lastModified() > lastModified) {
>                        lastModified = file.lastModified();
>                        return false;
>                }
>                return true;
>               }
>        }
>
>        @Override
>        public boolean remove(String arg0) {
>                return false;
>        }
>
>        public void setFileDir(String fileDir) {
>                this.fileDir = fileDir;
>        }
>
>        public String getFileDir() {
>                return fileDir;
>        }
> }
> --
> View this message in context: http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27357357.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

Claus Ibsen-2
On Sat, Jan 30, 2010 at 6:40 AM, vjintegrates <[hidden email]> wrote:

>
> I had tried this option it does not result in file being processed based on
> timestamp.
>
> readLock=changed
>
> I believe this attribute is used to to check if the file can be locked or
> not depending on last modified timestamp.
>
> Changed file were processed automatically in Camel 1.x, once I upgraded to
> 2.x it stopped working, therefore I decided to provide the implementation of
> IdempotentRepository
>

readLock=change works in similar way as Camel 1.x file component.
It uses both file length and timestamp to detect whether or not the
file "changes" over time.

You can enable TRACE logging at
org.apache.camel.component.file.strategy to see how it detects this.

The idea is that in 2.x to move or delete the processed file
AFTERWARDS it has been processed to avoid reading it again.
In 1.x this was not the case as it has this internal idempotent repo
as well. If you really want to use that you can do it in 2.x as well.
Beware that if you restart the server how are you going to know that
you have already processed the file before?


>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Have you tried the readLock=changed option?
>> http://camel.apache.org/file2.html
>>
>>
>> On Thu, Jan 28, 2010 at 4:01 PM, vjintegrates <[hidden email]>
>> wrote:
>>>
>>> I have added following implementation of IdempotentRepository to process
>>> a
>>> specific file from directory  when the file is modified. This is using
>>> Camel
>>> 2.x, Is there an alternative approach ?
>>>
>>> <from
>>> uri="file:src/data?noop=true&amp;idempotentRepository=#fileChanged&amp;delay=300000&amp;fileName=myfile.txt"/>
>>>
>>> <bean id="fileChanged" class="mypkg.FileChangedRepository">
>>>        <property name="fileDir" value="src/data" />
>>> </bean>
>>>
>>> public class FileChangedRepository implements
>>> IdempotentRepository<String>{
>>>
>>>        private String fileDir;
>>>        private long lastModified =0;
>>>
>>>        @Override
>>>        public boolean add(String arg0) {
>>>                return false;
>>>        }
>>>
>>>        @Override
>>>        public boolean confirm(String arg0) {
>>>                return true;
>>>        }
>>>
>>>        @Override
>>>        public boolean contains(String arg0) {
>>>                synchronized(this) {
>>>                File file = new File(fileDir + File.separator + arg0);
>>>
>>>                if (file.lastModified() > lastModified) {
>>>                        lastModified = file.lastModified();
>>>                        return false;
>>>                }
>>>                return true;
>>>               }
>>>        }
>>>
>>>        @Override
>>>        public boolean remove(String arg0) {
>>>                return false;
>>>        }
>>>
>>>        public void setFileDir(String fileDir) {
>>>                this.fileDir = fileDir;
>>>        }
>>>
>>>        public String getFileDir() {
>>>                return fileDir;
>>>        }
>>> }
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27357357.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27381562.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

vijayp
Tracing functionality was helpful.   "readLock=changed" moves the file to .camel/ dir, in our case we were trying keep the file in same location.  To avoid the move I used noop=true, and idempotent=false/true.

I saw output from following log statement even after setting the idempotent flag. That tells me setting idempotent did not work. Shouldn't the idempotent flag be respected ?

org/apache/camel/component/file/FileEndpoint.java

        // if noop=true then idempotent should also be configured
        if (isNoop() && !isIdempotentSet()) {
            log.info("Endpoint is configured with noop=true so forcing endpoint to be idempotent as well");
            setIdempotent(true);
        }

Use case is
1) process the file without moving it -
2) process the file only if it is modified -
3) it is okay to process same file again -



Claus Ibsen-2 wrote
On Sat, Jan 30, 2010 at 6:40 AM, vjintegrates <vijaypawnarkar@gmail.com> wrote:
>
> I had tried this option it does not result in file being processed based on
> timestamp.
>
> readLock=changed
>
> I believe this attribute is used to to check if the file can be locked or
> not depending on last modified timestamp.
>
> Changed file were processed automatically in Camel 1.x, once I upgraded to
> 2.x it stopped working, therefore I decided to provide the implementation of
> IdempotentRepository
>

readLock=change works in similar way as Camel 1.x file component.
It uses both file length and timestamp to detect whether or not the
file "changes" over time.

You can enable TRACE logging at
org.apache.camel.component.file.strategy to see how it detects this.

The idea is that in 2.x to move or delete the processed file
AFTERWARDS it has been processed to avoid reading it again.
In 1.x this was not the case as it has this internal idempotent repo
as well. If you really want to use that you can do it in 2.x as well.
Beware that if you restart the server how are you going to know that
you have already processed the file before?


>
>
> Claus Ibsen-2 wrote:
>>
>> Hi
>>
>> Have you tried the readLock=changed option?
>> http://camel.apache.org/file2.html
>>
>>
>> On Thu, Jan 28, 2010 at 4:01 PM, vjintegrates <vijaypawnarkar@gmail.com>
>> wrote:
>>>
>>> I have added following implementation of IdempotentRepository to process
>>> a
>>> specific file from directory  when the file is modified. This is using
>>> Camel
>>> 2.x, Is there an alternative approach ?
>>>
>>> <from
>>> uri="file:src/data?noop=true&idempotentRepository=#fileChanged&delay=300000&fileName=myfile.txt"/>
>>>
>>> <bean id="fileChanged" class="mypkg.FileChangedRepository">
>>>        <property name="fileDir" value="src/data" />
>>> </bean>
>>>
>>> public class FileChangedRepository implements
>>> IdempotentRepository<String>{
>>>
>>>        private String fileDir;
>>>        private long lastModified =0;
>>>
>>>        @Override
>>>        public boolean add(String arg0) {
>>>                return false;
>>>        }
>>>
>>>        @Override
>>>        public boolean confirm(String arg0) {
>>>                return true;
>>>        }
>>>
>>>        @Override
>>>        public boolean contains(String arg0) {
>>>                synchronized(this) {
>>>                File file = new File(fileDir + File.separator + arg0);
>>>
>>>                if (file.lastModified() > lastModified) {
>>>                        lastModified = file.lastModified();
>>>                        return false;
>>>                }
>>>                return true;
>>>               }
>>>        }
>>>
>>>        @Override
>>>        public boolean remove(String arg0) {
>>>                return false;
>>>        }
>>>
>>>        public void setFileDir(String fileDir) {
>>>                this.fileDir = fileDir;
>>>        }
>>>
>>>        public String getFileDir() {
>>>                return fileDir;
>>>        }
>>> }
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27357357.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27381562.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

Claus Ibsen-2
On Thu, Feb 4, 2010 at 1:22 PM, vjintegrates <[hidden email]> wrote:

>
> Tracing functionality was helpful.   "readLock=changed" moves the file to
> .camel/ dir, in our case we were trying keep the file in same location.  To
> avoid the move I used noop=true, and idempotent=false/true.
>
> I saw output from following log statement even after setting the idempotent
> flag. That tells me setting idempotent did not work. Shouldn't the
> idempotent flag be respected ?
>
> org/apache/camel/component/file/FileEndpoint.java
>
>        // if noop=true then idempotent should also be configured
>        if (isNoop() && !isIdempotentSet()) {
>            log.info("Endpoint is configured with noop=true so forcing
> endpoint to be idempotent as well");
>            setIdempotent(true);
>        }
>
> Use case is
> 1) process the file without moving it -
> 2) process the file only if it is modified -
> 3) it is okay to process same file again -
>
>

Can you post
- Camel version used
- Full route, especially endpoint URI

And try creating the route in Java DSL. I suspect XML may play trick
on idempotentSet


>
>
> Claus Ibsen-2 wrote:
>>
>> On Sat, Jan 30, 2010 at 6:40 AM, vjintegrates <[hidden email]>
>> wrote:
>>>
>>> I had tried this option it does not result in file being processed based
>>> on
>>> timestamp.
>>>
>>> readLock=changed
>>>
>>> I believe this attribute is used to to check if the file can be locked or
>>> not depending on last modified timestamp.
>>>
>>> Changed file were processed automatically in Camel 1.x, once I upgraded
>>> to
>>> 2.x it stopped working, therefore I decided to provide the implementation
>>> of
>>> IdempotentRepository
>>>
>>
>> readLock=change works in similar way as Camel 1.x file component.
>> It uses both file length and timestamp to detect whether or not the
>> file "changes" over time.
>>
>> You can enable TRACE logging at
>> org.apache.camel.component.file.strategy to see how it detects this.
>>
>> The idea is that in 2.x to move or delete the processed file
>> AFTERWARDS it has been processed to avoid reading it again.
>> In 1.x this was not the case as it has this internal idempotent repo
>> as well. If you really want to use that you can do it in 2.x as well.
>> Beware that if you restart the server how are you going to know that
>> you have already processed the file before?
>>
>>
>>>
>>>
>>> Claus Ibsen-2 wrote:
>>>>
>>>> Hi
>>>>
>>>> Have you tried the readLock=changed option?
>>>> http://camel.apache.org/file2.html
>>>>
>>>>
>>>> On Thu, Jan 28, 2010 at 4:01 PM, vjintegrates <[hidden email]>
>>>> wrote:
>>>>>
>>>>> I have added following implementation of IdempotentRepository to
>>>>> process
>>>>> a
>>>>> specific file from directory  when the file is modified. This is using
>>>>> Camel
>>>>> 2.x, Is there an alternative approach ?
>>>>>
>>>>> <from
>>>>> uri="file:src/data?noop=true&amp;idempotentRepository=#fileChanged&amp;delay=300000&amp;fileName=myfile.txt"/>
>>>>>
>>>>> <bean id="fileChanged" class="mypkg.FileChangedRepository">
>>>>>        <property name="fileDir" value="src/data" />
>>>>> </bean>
>>>>>
>>>>> public class FileChangedRepository implements
>>>>> IdempotentRepository<String>{
>>>>>
>>>>>        private String fileDir;
>>>>>        private long lastModified =0;
>>>>>
>>>>>        @Override
>>>>>        public boolean add(String arg0) {
>>>>>                return false;
>>>>>        }
>>>>>
>>>>>        @Override
>>>>>        public boolean confirm(String arg0) {
>>>>>                return true;
>>>>>        }
>>>>>
>>>>>        @Override
>>>>>        public boolean contains(String arg0) {
>>>>>                synchronized(this) {
>>>>>                File file = new File(fileDir + File.separator + arg0);
>>>>>
>>>>>                if (file.lastModified() > lastModified) {
>>>>>                        lastModified = file.lastModified();
>>>>>                        return false;
>>>>>                }
>>>>>                return true;
>>>>>               }
>>>>>        }
>>>>>
>>>>>        @Override
>>>>>        public boolean remove(String arg0) {
>>>>>                return false;
>>>>>        }
>>>>>
>>>>>        public void setFileDir(String fileDir) {
>>>>>                this.fileDir = fileDir;
>>>>>        }
>>>>>
>>>>>        public String getFileDir() {
>>>>>                return fileDir;
>>>>>        }
>>>>> }
>>>>> --
>>>>> View this message in context:
>>>>> http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27357357.html
>>>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> Apache Camel Committer
>>>>
>>>> Author of Camel in Action: http://www.manning.com/ibsen/
>>>> Open Source Integration: http://fusesource.com
>>>> Blog: http://davsclaus.blogspot.com/
>>>> Twitter: http://twitter.com/davsclaus
>>>>
>>>>
>>>
>>> --
>>> View this message in context:
>>> http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27381562.html
>>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>>
>>>
>>
>>
>>
>> --
>> Claus Ibsen
>> Apache Camel Committer
>>
>> Author of Camel in Action: http://www.manning.com/ibsen/
>> Open Source Integration: http://fusesource.com
>> Blog: http://davsclaus.blogspot.com/
>> Twitter: http://twitter.com/davsclaus
>>
>>
>
> --
> View this message in context: http://old.nabble.com/Identifying-and-processing-changed-file-when-noop%3Dtrue-tp27357357p27451896.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>



--
Claus Ibsen
Apache Camel Committer

Author of Camel in Action: http://www.manning.com/ibsen/
Open Source Integration: http://fusesource.com
Blog: http://davsclaus.blogspot.com/
Twitter: http://twitter.com/davsclaus
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

vijayp
This post was updated on .
I tried both XML and JavaDSL, results were consistent with XML and JavaDSL
XML

    <route>   
      <from uri="file:src/data?noop=true&amp;idempotent=false&amp;readLock=changed&amp;delay=60000&amp;fileName=Myfile.txt"/>
      <to uri="myPOJO"/>
    </route>
   
Java DSL

       CamelContext context = new DefaultCamelContext();
       context.addRoutes(new RouteBuilder() {
           public void configure() {
                  from("file:src/data?noop=true&idempotent=false&readLock=changed&delay=60000&fileName=Myfile.txt")
                  .process(new Processor(){
                                @Override
                              public void process(Exchange arg0)
                                    throws Exception {
                                        // TODO Auto-generated method stub
                                        logger.debug("Got the new file" + arg0.getIn().toString());
                                }
                        });
                   }
        });
        context.start();

With Camel 2.0.0
----------------------------------------------------
Result was the file was processed only first time and was never processed whether changed or not.

Upgraded and built it from Camel source 2.2-snapshot
----------------------------------------------------
Result was the File was processed every time whether changed or not. This seems to be correct behavior when idempotent=false, looks like a bug with Camel 2.0.0.

I went through the source code to discover that "readLock" property determines how file should be LOCKED, it does not determine whether file should be processed or not. That decision to process file is done by what are the values of "noop", "idempotent" flags along with FileIdempotentRepository. Since FileIdempotentRepository considers only filename/path to determine where the file was porcessed before or not.

In Use case is

 1) process the file without moving it -
 2) process the file only if it is modified -
 3) it is okay to process same file again during system restarts.

Providing an alternate implementation of  IdempotentRepository seem to provide solution.

 <from
 uri="file:src/data?noop=true&idempotentRepository=#fileChanged&delay=300000&fileName=myfile.txt"/>

 <bean id="fileChanged" class="mypkg.FileChangedRepository">
        <property name="fileDir" value="src/data" />
 </bean>

 public class FileChangedRepository implements
 IdempotentRepository<String>{

        private String fileDir;
        private long lastModified =0;

        @Override
        public boolean add(String arg0) {
                return false;
        }

        @Override
        public boolean confirm(String arg0) {
                return true;
        }

        @Override
        public boolean contains(String arg0) {
                synchronized(this) {
                File file = new File(fileDir + File.separator + arg0);

                if (file.lastModified() > lastModified) {
                        lastModified = file.lastModified();
                        return false;
                }
                return true;
               }
        }

        @Override
        public boolean remove(String arg0) {
                return false;
        }

        public void setFileDir(String fileDir) {
                this.fileDir = fileDir;
        }

        public String getFileDir() {
                return fileDir;
        }
 }
Claus Ibsen-2 wrote
On Thu, Feb 4, 2010 at 1:22 PM, vjintegrates <vijaypawnarkar@gmail.com> wrote:
>
> Tracing functionality was helpful.   "readLock=changed" moves the file to
> .camel/ dir, in our case we were trying keep the file in same location.  To
> avoid the move I used noop=true, and idempotent=false/true.
>
> I saw output from following log statement even after setting the idempotent
> flag. That tells me setting idempotent did not work. Shouldn't the
> idempotent flag be respected ?
>
> org/apache/camel/component/file/FileEndpoint.java
>
>        // if noop=true then idempotent should also be configured
>        if (isNoop() && !isIdempotentSet()) {
>            log.info("Endpoint is configured with noop=true so forcing
> endpoint to be idempotent as well");
>            setIdempotent(true);
>        }
>
> Use case is
> 1) process the file without moving it -
> 2) process the file only if it is modified -
> 3) it is okay to process same file again -
>
>

Can you post
- Camel version used
- Full route, especially endpoint URI

And try creating the route in Java DSL. I suspect XML may play trick
on idempotentSet


>
>
Reply | Threaded
Open this post in threaded view
|

Re: Identifying and processing changed file when noop=true

vijayp
Above message this did not show up on forum..