FilePolling

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

FilePolling

Ole Andreas Hegle
Hi

I am using Camel 1.0 (java 6) and have a file-component for polling
files from a folder.

However files only get polled when CamelContext is started, and not
later when new files are added to the folder I monitor.

I have configured it like this:
from("file:"+orderFolder).process(new OrderFileProcessor(context,
applicationEndpoint, adapter.getConfigurationFolder()));

(which when inserted becomes)
from("file:C:/folder").process(new Processor());

Does anybody know why files are only polled at startup, and not later?

Thanks in advance

-Ole Andreas

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

jstrachan
On 6/12/07, Ole Andreas Hegle <[hidden email]> wrote:
> Hi

Hi Ole!


> I am using Camel 1.0 (java 6) and have a file-component for polling
> files from a folder.
>
> However files only get polled when CamelContext is started, and not
> later when new files are added to the folder I monitor.

You start the context and don't stop it right?


> I have configured it like this:
> from("file:"+orderFolder).process(new OrderFileProcessor(context,
> applicationEndpoint, adapter.getConfigurationFolder()));
>
> (which when inserted becomes)
> from("file:C:/folder").process(new Processor());
>
> Does anybody know why files are only polled at startup, and not later?

The file consumer derived from the ScheduledPollConsumer which should
perform repeated timed polls (which are configurable on the endpoint
URI). Maybe if debug logging is enabled you'll see if the directory is
repeatedly pooled or not?

I guess a good test case could be for a route to be started, then a
new directory to be created after the first poll etc.

--
James
-------
http://macstrac.blogspot.com/
Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Ole Andreas Hegle
Hi James

Thank you for your answer.

I have just found the problem, the file-poller worked when i modified
the file, and it seems to me as the problem is located in
org.apache.camel.component.file.FileConsumer.java and the method

*protected* *void* pollFile(*final* File file) {
    *if* (file.exists() && file.lastModified() > lastPollTime) {
        *if* (isValidFile(file)) {
             processFile(file);
        }
    }
 }


My files are sent as attachment to a receiver by email, and then
imported to the system by saving the email-attachment in the folder
monitored by the FileConsumer. The modified-timestamp will be older than
the lastPollTime-timestamp, and because of this files will not be
polled. Are there any way to dissable the file.lastModified-method?
(I move the files to an archive folder when they are processed)

Perhaps I can have another component running and modifying the files to
change the last modified-timestamp on the files to make them pollable,
but I would rather not...

Anybody knows what I can do with this?

Regards Ole Andreas

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Rob Davies
ooh - school boy error - want to issue a patch? - if not I'll try and  
fix
On Jun 12, 2007, at 5:02 PM, Ole Andreas Hegle wrote:

> Hi James
>
> Thank you for your answer.
>
> I have just found the problem, the file-poller worked when i  
> modified the file, and it seems to me as the problem is located in
> org.apache.camel.component.file.FileConsumer.java and the method
>
> *protected* *void* pollFile(*final* File file) {
>    *if* (file.exists() && file.lastModified() > lastPollTime) {
>        *if* (isValidFile(file)) {
>             processFile(file);
>        }
>    }
> }
>
>
> My files are sent as attachment to a receiver by email, and then  
> imported to the system by saving the email-attachment in the folder  
> monitored by the FileConsumer. The modified-timestamp will be older  
> than the lastPollTime-timestamp, and because of this files will not  
> be polled. Are there any way to dissable the file.lastModified-method?
> (I move the files to an archive folder when they are processed)
>
> Perhaps I can have another component running and modifying the  
> files to change the last modified-timestamp on the files to make  
> them pollable, but I would rather not...
>
> Anybody knows what I can do with this?
>
> Regards Ole Andreas
>

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

jstrachan
On 6/12/07, Rob Davies <[hidden email]> wrote:
> ooh - school boy error - want to issue a patch? - if not I'll try and
> fix

Should we just trash the lastModified > lastPollTime test? Or make it optional?

James

> On Jun 12, 2007, at 5:02 PM, Ole Andreas Hegle wrote:
>
> > Hi James
> >
> > Thank you for your answer.
> >
> > I have just found the problem, the file-poller worked when i
> > modified the file, and it seems to me as the problem is located in
> > org.apache.camel.component.file.FileConsumer.java and the method
> >
> > *protected* *void* pollFile(*final* File file) {
> >    *if* (file.exists() && file.lastModified() > lastPollTime) {
> >        *if* (isValidFile(file)) {
> >             processFile(file);
> >        }
> >    }
> > }
> >
> >
> > My files are sent as attachment to a receiver by email, and then
> > imported to the system by saving the email-attachment in the folder
> > monitored by the FileConsumer. The modified-timestamp will be older
> > than the lastPollTime-timestamp, and because of this files will not
> > be polled. Are there any way to dissable the file.lastModified-method?
> > (I move the files to an archive folder when they are processed)
> >
> > Perhaps I can have another component running and modifying the
> > files to change the last modified-timestamp on the files to make
> > them pollable, but I would rather not...
> >
> > Anybody knows what I can do with this?
> >
> > Regards Ole Andreas
> >
>
>


--
James
-------
http://macstrac.blogspot.com/
Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Ole Andreas Hegle
Hi

Yes, to make it optional would be nice.

Other alternatives is to create an options for "delete processed files",
"move processed files" or to keep track of all files in the polled
folder to prevent files beeing polled twice, unless they are modified.

-Ole Andreas


James Strachan wrote:

> On 6/12/07, Rob Davies <[hidden email]> wrote:
>> ooh - school boy error - want to issue a patch? - if not I'll try and
>> fix
>
> Should we just trash the lastModified > lastPollTime test? Or make it
> optional?
>
> James
>
>> On Jun 12, 2007, at 5:02 PM, Ole Andreas Hegle wrote:
>>
>> > Hi James
>> >
>> > Thank you for your answer.
>> >
>> > I have just found the problem, the file-poller worked when i
>> > modified the file, and it seems to me as the problem is located in
>> > org.apache.camel.component.file.FileConsumer.java and the method
>> >
>> > *protected* *void* pollFile(*final* File file) {
>> >    *if* (file.exists() && file.lastModified() > lastPollTime) {
>> >        *if* (isValidFile(file)) {
>> >             processFile(file);
>> >        }
>> >    }
>> > }
>> >
>> >
>> > My files are sent as attachment to a receiver by email, and then
>> > imported to the system by saving the email-attachment in the folder
>> > monitored by the FileConsumer. The modified-timestamp will be older
>> > than the lastPollTime-timestamp, and because of this files will not
>> > be polled. Are there any way to dissable the file.lastModified-method?
>> > (I move the files to an archive folder when they are processed)
>> >
>> > Perhaps I can have another component running and modifying the
>> > files to change the last modified-timestamp on the files to make
>> > them pollable, but I would rather not...
>> >
>> > Anybody knows what I can do with this?
>> >
>> > Regards Ole Andreas
>> >
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Hiram Chirino
On 6/12/07, Ole Andreas Hegle <[hidden email]> wrote:
> Hi
>
> Yes, to make it optional would be nice.
>
> Other alternatives is to create an options for "delete processed files",
> "move processed files" or to keep track of all files in the polled
> folder to prevent files beeing polled twice, unless they are modified.
>

Those 3 strategies make lots of sense.. want to take a stab at
implementing them ;-)

> -Ole Andreas
>
>
> James Strachan wrote:
> > On 6/12/07, Rob Davies <[hidden email]> wrote:
> >> ooh - school boy error - want to issue a patch? - if not I'll try and
> >> fix
> >
> > Should we just trash the lastModified > lastPollTime test? Or make it
> > optional?
> >
> > James
> >
> >> On Jun 12, 2007, at 5:02 PM, Ole Andreas Hegle wrote:
> >>
> >> > Hi James
> >> >
> >> > Thank you for your answer.
> >> >
> >> > I have just found the problem, the file-poller worked when i
> >> > modified the file, and it seems to me as the problem is located in
> >> > org.apache.camel.component.file.FileConsumer.java and the method
> >> >
> >> > *protected* *void* pollFile(*final* File file) {
> >> >    *if* (file.exists() && file.lastModified() > lastPollTime) {
> >> >        *if* (isValidFile(file)) {
> >> >             processFile(file);
> >> >        }
> >> >    }
> >> > }
> >> >
> >> >
> >> > My files are sent as attachment to a receiver by email, and then
> >> > imported to the system by saving the email-attachment in the folder
> >> > monitored by the FileConsumer. The modified-timestamp will be older
> >> > than the lastPollTime-timestamp, and because of this files will not
> >> > be polled. Are there any way to dissable the file.lastModified-method?
> >> > (I move the files to an archive folder when they are processed)
> >> >
> >> > Perhaps I can have another component running and modifying the
> >> > files to change the last modified-timestamp on the files to make
> >> > them pollable, but I would rather not...
> >> >
> >> > Anybody knows what I can do with this?
> >> >
> >> > Regards Ole Andreas
> >> >
> >>
> >>
> >
> >
>
>


--
Regards,
Hiram

Blog: http://hiramchirino.com
Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Ole Andreas Hegle
Hi

I need at least functionality to disable the last-modified vs. polltime
test, så I can create a fix for this.

How/where do I submit the patch when it is ready?

-Ole Andreas

Hiram Chirino wrote:

> On 6/12/07, Ole Andreas Hegle <[hidden email]> wrote:
>> Hi
>>
>> Yes, to make it optional would be nice.
>>
>> Other alternatives is to create an options for "delete processed files",
>> "move processed files" or to keep track of all files in the polled
>> folder to prevent files beeing polled twice, unless they are modified.
>>
>
> Those 3 strategies make lots of sense.. want to take a stab at
> implementing them ;-)
>
>> -Ole Andreas
>>
>>
>> James Strachan wrote:
>> > On 6/12/07, Rob Davies <[hidden email]> wrote:
>> >> ooh - school boy error - want to issue a patch? - if not I'll try and
>> >> fix
>> >
>> > Should we just trash the lastModified > lastPollTime test? Or make it
>> > optional?
>> >
>> > James
>> >
>> >> On Jun 12, 2007, at 5:02 PM, Ole Andreas Hegle wrote:
>> >>
>> >> > Hi James
>> >> >
>> >> > Thank you for your answer.
>> >> >
>> >> > I have just found the problem, the file-poller worked when i
>> >> > modified the file, and it seems to me as the problem is located in
>> >> > org.apache.camel.component.file.FileConsumer.java and the method
>> >> >
>> >> > *protected* *void* pollFile(*final* File file) {
>> >> >    *if* (file.exists() && file.lastModified() > lastPollTime) {
>> >> >        *if* (isValidFile(file)) {
>> >> >             processFile(file);
>> >> >        }
>> >> >    }
>> >> > }
>> >> >
>> >> >
>> >> > My files are sent as attachment to a receiver by email, and then
>> >> > imported to the system by saving the email-attachment in the folder
>> >> > monitored by the FileConsumer. The modified-timestamp will be older
>> >> > than the lastPollTime-timestamp, and because of this files will not
>> >> > be polled. Are there any way to dissable the
>> file.lastModified-method?
>> >> > (I move the files to an archive folder when they are processed)
>> >> >
>> >> > Perhaps I can have another component running and modifying the
>> >> > files to change the last modified-timestamp on the files to make
>> >> > them pollable, but I would rather not...
>> >> >
>> >> > Anybody knows what I can do with this?
>> >> >
>> >> > Regards Ole Andreas
>> >> >
>> >>
>> >>
>> >
>> >
>>
>>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

jstrachan
On 6/12/07, Ole Andreas Hegle <[hidden email]> wrote:
> Hi
>
> I need at least functionality to disable the last-modified vs. polltime
> test, så I can create a fix for this.

Awesome! :)

> How/where do I submit the patch when it is ready?

Here's details

http://activemq.apache.org/camel/contributing.html

a patch file is ideal; if you're having trouble making one, just
attach the code to a JIRA and we can take it from there. Many thanks
Ole!

--
James
-------
http://macstrac.blogspot.com/
Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

Ole Andreas Hegle
Hi There

I have discovered some issues with the file-component, but I need to
clear up one thing here.

Are there any reason why the ScheduledPollEndpoint use the
"consumer."-prefix on options which go to the ScheduledPollConsumer?  
(initialDelay, delay and useFixedDelay).
As far as I can se, no other endpoints/consumers use a
"consumer."-prefix for sorting-attributes. Shall we keep the
"consumer."-prefix, or shall we move the attributes from
ScheduledPollConsumer to ScheduledPollEndpoint? What design guidelines
do you prefer? (If we keep it in the consumer, we must update the
user-manual on the web)

Regards. Ole Andreas



James Strachan wrote:

> On 6/12/07, Ole Andreas Hegle <[hidden email]> wrote:
>> Hi
>>
>> I need at least functionality to disable the last-modified vs. polltime
>> test, så I can create a fix for this.
>
> Awesome! :)
>
>> How/where do I submit the patch when it is ready?
>
> Here's details
>
> http://activemq.apache.org/camel/contributing.html
>
> a patch file is ideal; if you're having trouble making one, just
> attach the code to a JIRA and we can take it from there. Many thanks
> Ole!
>

Reply | Threaded
Open this post in threaded view
|

Re: FilePolling

jstrachan
On 6/15/07, Ole Andreas Hegle <[hidden email]> wrote:

> Hi There
>
> I have discovered some issues with the file-component, but I need to
> clear up one thing here.
>
> Are there any reason why the ScheduledPollEndpoint use the
> "consumer."-prefix on options which go to the ScheduledPollConsumer?
> (initialDelay, delay and useFixedDelay).
> As far as I can se, no other endpoints/consumers use a
> "consumer."-prefix for sorting-attributes. Shall we keep the
> "consumer."-prefix, or shall we move the attributes from
> ScheduledPollConsumer to ScheduledPollEndpoint? What design guidelines
> do you prefer? (If we keep it in the consumer, we must update the
> user-manual on the web)

The reason I separated them out like that was in case the endpoint or
producer had separate properties. Though as an end user, I guess it
kinda sucks :).

The main issue is you create & validate the endpoint; then later on
you create & validate the consumer; which is why I kinda split them up
a bit (so that at each step we can validate the properties used).

It'd be cleaner for end users if we just had a flat properties thing;
but that'd make the possible validation code a tad more complex
(having to check that any remaining properties on an endpoint
parameter map really are destined for a producer/consumer).

If someone can figure out a neat way to solve that so we can add
properties for the producer, consumer or endpoint in a simple way that
also can validate; am really happy to trash the prefix stuff (it was
just the easiest way to get started :)

--
James
-------
http://macstrac.blogspot.com/