No way to remove FTP/SFTP files?

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

No way to remove FTP/SFTP files?

Drew McAuliffe
Am I crazy or is there no way that the FTP/SFTP consumers do anything to a source file once they've read it? There doesn't appear to be any delete or move option like with the File consumer. What possible use case would that support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will continuously re-read files, never stopping?

It appears that there's a JIRA issue to add support for something similar to a file renaming strategy, but it doesn't look like it's slated for anything in 1.4.
Reply | Threaded
Open this post in threaded view
|

Re: No way to remove FTP/SFTP files?

gertv
Drew,


No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
this.  They store the lastPollTimestamp internally and compare that to
the file's timestamps to determine if a file still needs to be
processed.  However, the lastPollTime isn't being persisted anywhere, so
restarting the JVM will result in re-reading the files.

If you want, you can always provide a patch for the file-handling
strategies you want to use for your project.  Let us know if you need
any help with that...  Any (even partial) solution for this issue would
be more than welcome.

If you are using Camel inside ServiceMix, you can also use ServiceMix's
FTP poller component as a (temporary) workaround.  That one does already
support most forms of file-handling (delete, move, ...).


Regards,

Gert



Drew McAuliffe wrote:

> Am I crazy or is there no way that the FTP/SFTP consumers do anything to a
> source file once they've read it? There doesn't appear to be any delete or
> move option like with the File consumer. What possible use case would that
> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will
> continuously re-read files, never stopping?
>
> It appears that there's a JIRA issue to add support for something similar to
> a file renaming strategy, but it doesn't look like it's slated for anything
> in 1.4.
>  

Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Claus Ibsen
Hi

And voting for the tickets also helps the team decide which issues we should tackle first.

Personally I do think file based components needs an overhaul in Camel to improve their standards as this is IMHO still one of the most common integration techniques in my daily fields of work.

Gert good point about that ServiceMix FTP component. Good spot for code resuse.



Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Gert Vanthienen [mailto:[hidden email]]
Sent: 3. juni 2008 04:42
To: [hidden email]
Subject: Re: No way to remove FTP/SFTP files?

Drew,


No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
this.  They store the lastPollTimestamp internally and compare that to
the file's timestamps to determine if a file still needs to be
processed.  However, the lastPollTime isn't being persisted anywhere, so
restarting the JVM will result in re-reading the files.

If you want, you can always provide a patch for the file-handling
strategies you want to use for your project.  Let us know if you need
any help with that...  Any (even partial) solution for this issue would
be more than welcome.

If you are using Camel inside ServiceMix, you can also use ServiceMix's
FTP poller component as a (temporary) workaround.  That one does already
support most forms of file-handling (delete, move, ...).


Regards,

Gert



Drew McAuliffe wrote:

> Am I crazy or is there no way that the FTP/SFTP consumers do anything to a
> source file once they've read it? There doesn't appear to be any delete or
> move option like with the File consumer. What possible use case would that
> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will
> continuously re-read files, never stopping?
>
> It appears that there's a JIRA issue to add support for something similar to
> a file renaming strategy, but it doesn't look like it's slated for anything
> in 1.4.
>  

Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Claus Ibsen
Hi

Added ticket: CAMEL-570 to track this issue.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk

-----Original Message-----
From: Claus Ibsen [mailto:[hidden email]]
Sent: 3. juni 2008 05:50
To: [hidden email]
Subject: RE: No way to remove FTP/SFTP files?

Hi

And voting for the tickets also helps the team decide which issues we should tackle first.

Personally I do think file based components needs an overhaul in Camel to improve their standards as this is IMHO still one of the most common integration techniques in my daily fields of work.

Gert good point about that ServiceMix FTP component. Good spot for code resuse.



Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Gert Vanthienen [mailto:[hidden email]]
Sent: 3. juni 2008 04:42
To: [hidden email]
Subject: Re: No way to remove FTP/SFTP files?

Drew,


No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
this.  They store the lastPollTimestamp internally and compare that to
the file's timestamps to determine if a file still needs to be
processed.  However, the lastPollTime isn't being persisted anywhere, so
restarting the JVM will result in re-reading the files.

If you want, you can always provide a patch for the file-handling
strategies you want to use for your project.  Let us know if you need
any help with that...  Any (even partial) solution for this issue would
be more than welcome.

If you are using Camel inside ServiceMix, you can also use ServiceMix's
FTP poller component as a (temporary) workaround.  That one does already
support most forms of file-handling (delete, move, ...).


Regards,

Gert



Drew McAuliffe wrote:

> Am I crazy or is there no way that the FTP/SFTP consumers do anything to a
> source file once they've read it? There doesn't appear to be any delete or
> move option like with the File consumer. What possible use case would that
> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will
> continuously re-read files, never stopping?
>
> It appears that there's a JIRA issue to add support for something similar to
> a file renaming strategy, but it doesn't look like it's slated for anything
> in 1.4.
>  

Reply | Threaded
Open this post in threaded view
|

Re: No way to remove FTP/SFTP files?

Drew McAuliffe
In reply to this post by gertv
I've been working on some sort of patch and wanted to see if the following would work. Note that it's based on recent changes to the file processor for 1.4, so that the commit on file process strategy gets called even if there's an error on the processing.

So the quick question is, whether it's simply a matter of replacing this code in the Ftp and Sftp consumer components' "pollFile" method:

getProcessor().process(exchange);

...with this, modeled after the file component (note that I only have comments so far related to file processing):

                // begin file process strategy
                //final FileProcessStrategy processStrategy = endpoint.getFileStrategy();
                //if(processStrategy.begin(...
                // process async, with callback
                getAsyncProcessor().process(exchange, new AsyncCallback(){
                public void done(boolean sync) {
                        boolean failed = exchange.isFailed();
                        boolean handled = DeadLetterChannel.isFailureHandled(exchange);
                       
                        if (LOG.isDebugEnabled()) {
                            LOG.debug("Done processing file: " + ftpFile + ". Status is: " + (failed ? "failed: " + failed + ", handled by failure processor: " + handled : "OK"));
                        }

                        if (!failed || handled) {
                            // commit the file strategy if there was no failure or already handled by the DeadLetterChannel
                            //processStrategyCommit(...
                        } else if (failed && !handled) {
                            // there was an exception but it was not handled by the DeadLetterChannel
                            handleException(exchange.getException());
                        }
                }
                });


The big question is whether or not the switch to an asynch call for processing the message will have any weird effect on the Ftp or Sftp clients.

Of course, the easy way to handle this across the board would be to replace the File, Ftp, and Sftp components with a Vfs component....:)

Gert Vanthienen wrote
Drew,


No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
this.  They store the lastPollTimestamp internally and compare that to
the file's timestamps to determine if a file still needs to be
processed.  However, the lastPollTime isn't being persisted anywhere, so
restarting the JVM will result in re-reading the files.

If you want, you can always provide a patch for the file-handling
strategies you want to use for your project.  Let us know if you need
any help with that...  Any (even partial) solution for this issue would
be more than welcome.

If you are using Camel inside ServiceMix, you can also use ServiceMix's
FTP poller component as a (temporary) workaround.  That one does already
support most forms of file-handling (delete, move, ...).


Regards,

Gert



Drew McAuliffe wrote:
> Am I crazy or is there no way that the FTP/SFTP consumers do anything to a
> source file once they've read it? There doesn't appear to be any delete or
> move option like with the File consumer. What possible use case would that
> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will
> continuously re-read files, never stopping?
>
> It appears that there's a JIRA issue to add support for something similar to
> a file renaming strategy, but it doesn't look like it's slated for anything
> in 1.4.
>  



-----
---
Gert Vanthienen
http://www.anova.be
Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Drew McAuliffe
In reply to this post by Claus Ibsen
Just as an FYI, I've hacked the following (after adding an "autoDelete" property to the endpoint configuration. This is in "pollFile" in FtpConsumer. It's a little staggered for debugging purposes, and I'm sure the exception throwing isn't ideal, but it seems to work. This is based somewhat on the Mule FTP and SFTP components. I'm working on the SFTP version next. While I think a FileProcessStrategy approach similar to the File component might be nice, for now an autodelete option works just fine (I delete the files read from SFTP after moving them to local storage, where I can handle them with proper backup after reading). Again, I think if you're going to go through the effort of implementing some sort of FileProcessStrategy thing, it would be better to do it once using a VFS component instead of splitting it between FTP and SFTP because of the different clients.

                getAsyncProcessor().process(exchange, new AsyncCallback(){
                public void done(boolean sync) {
                        boolean failed = exchange.isFailed();
                        boolean handled = DeadLetterChannel.isFailureHandled(exchange);
                       
                        if (LOG.isDebugEnabled()) {
                            LOG.debug("Done processing file: " + ftpFile + ". Status is: " + (failed ? "failed: " + failed + ", handled by failure processor: " + handled : "OK"));
                        }

                        if (!failed || handled) {
                        // if autodelete, then remove the source file
                        if (endpoint.getConfiguration().isAutoDelete()){
                        try {
                        String toDelete = getFullFileName(ftpFile);
                                                                        boolean deleted = client.deleteFile(toDelete);
                                                                        if (! deleted)
                                                                                throw new IOException(MessageFormat.format("Could not delete remote file at path {0}. Ftp error:{1}",
                                                                                        new Object[]{ftpFile.getName(), client.getReplyCode()}));
                                                                       
                                                                } catch (IOException e) {
                                                                        throw new RuntimeException(e.getMessage(), e);
                                                                }
                        }
                        } else if (failed && !handled) {
                            // there was an exception but it was not handled by the DeadLetterChannel
                            handleException(exchange.getException());
                        }
                }
                });
 

Claus Ibsen wrote
Hi

Added ticket: CAMEL-570 to track this issue.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk

-----Original Message-----
From: Claus Ibsen [mailto:ci@silverbullet.dk]
Sent: 3. juni 2008 05:50
To: camel-user@activemq.apache.org
Subject: RE: No way to remove FTP/SFTP files?

Hi

And voting for the tickets also helps the team decide which issues we should tackle first.

Personally I do think file based components needs an overhaul in Camel to improve their standards as this is IMHO still one of the most common integration techniques in my daily fields of work.

Gert good point about that ServiceMix FTP component. Good spot for code resuse.



Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Gert Vanthienen [mailto:gert.vanthienen@skynet.be]
Sent: 3. juni 2008 04:42
To: camel-user@activemq.apache.org
Subject: Re: No way to remove FTP/SFTP files?

Drew,


No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
this.  They store the lastPollTimestamp internally and compare that to
the file's timestamps to determine if a file still needs to be
processed.  However, the lastPollTime isn't being persisted anywhere, so
restarting the JVM will result in re-reading the files.

If you want, you can always provide a patch for the file-handling
strategies you want to use for your project.  Let us know if you need
any help with that...  Any (even partial) solution for this issue would
be more than welcome.

If you are using Camel inside ServiceMix, you can also use ServiceMix's
FTP poller component as a (temporary) workaround.  That one does already
support most forms of file-handling (delete, move, ...).


Regards,

Gert



Drew McAuliffe wrote:
> Am I crazy or is there no way that the FTP/SFTP consumers do anything to a
> source file once they've read it? There doesn't appear to be any delete or
> move option like with the File consumer. What possible use case would that
> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer will
> continuously re-read files, never stopping?
>
> It appears that there's a JIRA issue to add support for something similar to
> a file renaming strategy, but it doesn't look like it's slated for anything
> in 1.4.
>  
Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Claus Ibsen
Hi Drew

Great work. Keep it up and I will later commit it into Camel.
When you think its ready then please submit it as a patch to CAMEL-570.

Camel does really need this feature to remove consumed FTP files. Transferring messages using FTP is still a very common use case in real life.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:[hidden email]]
Sent: 6. juni 2008 07:35
To: [hidden email]
Subject: RE: No way to remove FTP/SFTP files?


Just as an FYI, I've hacked the following (after adding an "autoDelete"
property to the endpoint configuration. This is in "pollFile" in
FtpConsumer. It's a little staggered for debugging purposes, and I'm sure
the exception throwing isn't ideal, but it seems to work. This is based
somewhat on the Mule FTP and SFTP components. I'm working on the SFTP
version next. While I think a FileProcessStrategy approach similar to the
File component might be nice, for now an autodelete option works just fine
(I delete the files read from SFTP after moving them to local storage, where
I can handle them with proper backup after reading). Again, I think if
you're going to go through the effort of implementing some sort of
FileProcessStrategy thing, it would be better to do it once using a VFS
component instead of splitting it between FTP and SFTP because of the
different clients.

                getAsyncProcessor().process(exchange, new AsyncCallback(){
                public void done(boolean sync) {
                        boolean failed = exchange.isFailed();
                        boolean handled =
DeadLetterChannel.isFailureHandled(exchange);
                       
                        if (LOG.isDebugEnabled()) {
                            LOG.debug("Done processing file: " + ftpFile +
". Status is: " + (failed ? "failed: " + failed + ", handled by failure
processor: " + handled : "OK"));
                        }

                        if (!failed || handled) {
                        // if autodelete, then remove the source file
                        if (endpoint.getConfiguration().isAutoDelete()){
                        try {
                        String toDelete = getFullFileName(ftpFile);
                                                                        boolean deleted = client.deleteFile(toDelete);
                                                                        if (! deleted)
                                                                                throw new IOException(MessageFormat.format("Could not delete
remote file at path {0}. Ftp error:{1}",
                                                                                        new Object[]{ftpFile.getName(), client.getReplyCode()}));
                                                                       
                                                                } catch (IOException e) {
                                                                        throw new RuntimeException(e.getMessage(), e);
                                                                }
                        }
                        } else if (failed && !handled) {
                            // there was an exception but it was not handled
by the DeadLetterChannel
                            handleException(exchange.getException());
                        }
                }
                });
 


Claus Ibsen wrote:

>
> Hi
>
> Added ticket: CAMEL-570 to track this issue.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Claus Ibsen [mailto:[hidden email]]
> Sent: 3. juni 2008 05:50
> To: [hidden email]
> Subject: RE: No way to remove FTP/SFTP files?
>
> Hi
>
> And voting for the tickets also helps the team decide which issues we
> should tackle first.
>
> Personally I do think file based components needs an overhaul in Camel to
> improve their standards as this is IMHO still one of the most common
> integration techniques in my daily fields of work.
>
> Gert good point about that ServiceMix FTP component. Good spot for code
> resuse.
>
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
> -----Original Message-----
> From: Gert Vanthienen [mailto:[hidden email]]
> Sent: 3. juni 2008 04:42
> To: [hidden email]
> Subject: Re: No way to remove FTP/SFTP files?
>
> Drew,
>
>
> No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
> this.  They store the lastPollTimestamp internally and compare that to
> the file's timestamps to determine if a file still needs to be
> processed.  However, the lastPollTime isn't being persisted anywhere, so
> restarting the JVM will result in re-reading the files.
>
> If you want, you can always provide a patch for the file-handling
> strategies you want to use for your project.  Let us know if you need
> any help with that...  Any (even partial) solution for this issue would
> be more than welcome.
>
> If you are using Camel inside ServiceMix, you can also use ServiceMix's
> FTP poller component as a (temporary) workaround.  That one does already
> support most forms of file-handling (delete, move, ...).
>
>
> Regards,
>
> Gert
>
>
>
> Drew McAuliffe wrote:
>> Am I crazy or is there no way that the FTP/SFTP consumers do anything to
>> a
>> source file once they've read it? There doesn't appear to be any delete
>> or
>> move option like with the File consumer. What possible use case would
>> that
>> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer
>> will
>> continuously re-read files, never stopping?
>>
>> It appears that there's a JIRA issue to add support for something similar
>> to
>> a file renaming strategy, but it doesn't look like it's slated for
>> anything
>> in 1.4.
>>  
>
>
>

--
View this message in context: http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17684963.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Drew McAuliffe
Before I submit it, any ideas on how to handle the exception throwing in the processing? I'm just throwing a RuntimeException because I didn't know what the normal Camel procedure was. I have to trap the IOException to adhere to the interface of the async processor.

Also, are there any repercussions for doing this in an async processor block, as opposed to the way the component handles it now (synchronously with a call directly to "process")?


Claus Ibsen wrote
Hi Drew

Great work. Keep it up and I will later commit it into Camel.
When you think its ready then please submit it as a patch to CAMEL-570.

Camel does really need this feature to remove consumed FTP files. Transferring messages using FTP is still a very common use case in real life.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk

-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@gmail.com]
Sent: 6. juni 2008 07:35
To: camel-user@activemq.apache.org
Subject: RE: No way to remove FTP/SFTP files?


Just as an FYI, I've hacked the following (after adding an "autoDelete"
property to the endpoint configuration. This is in "pollFile" in
FtpConsumer. It's a little staggered for debugging purposes, and I'm sure
the exception throwing isn't ideal, but it seems to work. This is based
somewhat on the Mule FTP and SFTP components. I'm working on the SFTP
version next. While I think a FileProcessStrategy approach similar to the
File component might be nice, for now an autodelete option works just fine
(I delete the files read from SFTP after moving them to local storage, where
I can handle them with proper backup after reading). Again, I think if
you're going to go through the effort of implementing some sort of
FileProcessStrategy thing, it would be better to do it once using a VFS
component instead of splitting it between FTP and SFTP because of the
different clients.

                getAsyncProcessor().process(exchange, new AsyncCallback(){
                public void done(boolean sync) {
                        boolean failed = exchange.isFailed();
                        boolean handled =
DeadLetterChannel.isFailureHandled(exchange);
                       
                        if (LOG.isDebugEnabled()) {
                            LOG.debug("Done processing file: " + ftpFile +
". Status is: " + (failed ? "failed: " + failed + ", handled by failure
processor: " + handled : "OK"));
                        }

                        if (!failed || handled) {
                        // if autodelete, then remove the source file
                        if (endpoint.getConfiguration().isAutoDelete()){
                        try {
                        String toDelete = getFullFileName(ftpFile);
                                                                        boolean deleted = client.deleteFile(toDelete);
                                                                        if (! deleted)
                                                                                throw new IOException(MessageFormat.format("Could not delete
remote file at path {0}. Ftp error:{1}",
                                                                                        new Object[]{ftpFile.getName(), client.getReplyCode()}));
                                                                       
                                                                } catch (IOException e) {
                                                                        throw new RuntimeException(e.getMessage(), e);
                                                                }
                        }
                        } else if (failed && !handled) {
                            // there was an exception but it was not handled
by the DeadLetterChannel
                            handleException(exchange.getException());
                        }
                }
                });
 


Claus Ibsen wrote:
>
> Hi
>
> Added ticket: CAMEL-570 to track this issue.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Claus Ibsen [mailto:ci@silverbullet.dk]
> Sent: 3. juni 2008 05:50
> To: camel-user@activemq.apache.org
> Subject: RE: No way to remove FTP/SFTP files?
>
> Hi
>
> And voting for the tickets also helps the team decide which issues we
> should tackle first.
>
> Personally I do think file based components needs an overhaul in Camel to
> improve their standards as this is IMHO still one of the most common
> integration techniques in my daily fields of work.
>
> Gert good point about that ServiceMix FTP component. Good spot for code
> resuse.
>
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
> -----Original Message-----
> From: Gert Vanthienen [mailto:gert.vanthienen@skynet.be]
> Sent: 3. juni 2008 04:42
> To: camel-user@activemq.apache.org
> Subject: Re: No way to remove FTP/SFTP files?
>
> Drew,
>
>
> No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
> this.  They store the lastPollTimestamp internally and compare that to
> the file's timestamps to determine if a file still needs to be
> processed.  However, the lastPollTime isn't being persisted anywhere, so
> restarting the JVM will result in re-reading the files.
>
> If you want, you can always provide a patch for the file-handling
> strategies you want to use for your project.  Let us know if you need
> any help with that...  Any (even partial) solution for this issue would
> be more than welcome.
>
> If you are using Camel inside ServiceMix, you can also use ServiceMix's
> FTP poller component as a (temporary) workaround.  That one does already
> support most forms of file-handling (delete, move, ...).
>
>
> Regards,
>
> Gert
>
>
>
> Drew McAuliffe wrote:
>> Am I crazy or is there no way that the FTP/SFTP consumers do anything to
>> a
>> source file once they've read it? There doesn't appear to be any delete
>> or
>> move option like with the File consumer. What possible use case would
>> that
>> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer
>> will
>> continuously re-read files, never stopping?
>>
>> It appears that there's a JIRA issue to add support for something similar
>> to
>> a file renaming strategy, but it doesn't look like it's slated for
>> anything
>> in 1.4.
>>  
>
>
>

--
View this message in context: http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17684963.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Claus Ibsen
Hi Drew

Wrapping into a runtime exception is okay, but try to find a suitable exception in camel-core.

I do think that Camel could be improved here to adhere to a more common/strict exception hieracy - but that can be a goal for Camel 2.0 where the API can be changed slightly.

The FTPConsumer will have it's exception handled by the ScheduledPollConsumer#run.


About the async, well I do not think so at current time. Is there any real
life use-case for this requirements?

Let's keep the task simple to remedy the problem with the lack of a delete=true|false option on the component.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Drew McAuliffe [mailto:[hidden email]]
Sent: 6. juni 2008 22:39
To: [hidden email]
Subject: RE: No way to remove FTP/SFTP files?


Before I submit it, any ideas on how to handle the exception throwing in the
processing? I'm just throwing a RuntimeException because I didn't know what
the normal Camel procedure was. I have to trap the IOException to adhere to
the interface of the async processor.

Also, are there any repercussions for doing this in an async processor
block, as opposed to the way the component handles it now (synchronously
with a call directly to "process")?



Claus Ibsen wrote:

>
> Hi Drew
>
> Great work. Keep it up and I will later commit it into Camel.
> When you think its ready then please submit it as a patch to CAMEL-570.
>
> Camel does really need this feature to remove consumed FTP files.
> Transferring messages using FTP is still a very common use case in real
> life.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Drew McAuliffe [mailto:[hidden email]]
> Sent: 6. juni 2008 07:35
> To: [hidden email]
> Subject: RE: No way to remove FTP/SFTP files?
>
>
> Just as an FYI, I've hacked the following (after adding an "autoDelete"
> property to the endpoint configuration. This is in "pollFile" in
> FtpConsumer. It's a little staggered for debugging purposes, and I'm sure
> the exception throwing isn't ideal, but it seems to work. This is based
> somewhat on the Mule FTP and SFTP components. I'm working on the SFTP
> version next. While I think a FileProcessStrategy approach similar to the
> File component might be nice, for now an autodelete option works just fine
> (I delete the files read from SFTP after moving them to local storage,
> where
> I can handle them with proper backup after reading). Again, I think if
> you're going to go through the effort of implementing some sort of
> FileProcessStrategy thing, it would be better to do it once using a VFS
> component instead of splitting it between FTP and SFTP because of the
> different clients.
>
>                 getAsyncProcessor().process(exchange, new AsyncCallback(){
>                 public void done(boolean sync) {
>                         boolean failed = exchange.isFailed();
>                         boolean handled =
> DeadLetterChannel.isFailureHandled(exchange);
>                        
>                         if (LOG.isDebugEnabled()) {
>                             LOG.debug("Done processing file: " + ftpFile +
> ". Status is: " + (failed ? "failed: " + failed + ", handled by failure
> processor: " + handled : "OK"));
>                         }
>
>                         if (!failed || handled) {
>                         // if autodelete, then remove the source file
>                         if (endpoint.getConfiguration().isAutoDelete()){
>                         try {
>                         String toDelete = getFullFileName(ftpFile);
> boolean deleted = client.deleteFile(toDelete);
> if (! deleted)
> throw new IOException(MessageFormat.format("Could not delete
> remote file at path {0}. Ftp error:{1}",
> new Object[]{ftpFile.getName(), client.getReplyCode()}));
>
> } catch (IOException e) {
> throw new RuntimeException(e.getMessage(), e);
> }
>                         }
>                         } else if (failed && !handled) {
>                             // there was an exception but it was not
> handled
> by the DeadLetterChannel
>                             handleException(exchange.getException());
>                         }
>                 }
>                 });
>  
>
>
> Claus Ibsen wrote:
>>
>> Hi
>>
>> Added ticket: CAMEL-570 to track this issue.
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. +45 2962 7576
>> Web: www.silverbullet.dk
>>
>> -----Original Message-----
>> From: Claus Ibsen [mailto:[hidden email]]
>> Sent: 3. juni 2008 05:50
>> To: [hidden email]
>> Subject: RE: No way to remove FTP/SFTP files?
>>
>> Hi
>>
>> And voting for the tickets also helps the team decide which issues we
>> should tackle first.
>>
>> Personally I do think file based components needs an overhaul in Camel to
>> improve their standards as this is IMHO still one of the most common
>> integration techniques in my daily fields of work.
>>
>> Gert good point about that ServiceMix FTP component. Good spot for code
>> resuse.
>>
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. +45 2962 7576
>> Web: www.silverbullet.dk
>> -----Original Message-----
>> From: Gert Vanthienen [mailto:[hidden email]]
>> Sent: 3. juni 2008 04:42
>> To: [hidden email]
>> Subject: Re: No way to remove FTP/SFTP files?
>>
>> Drew,
>>
>>
>> No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
>> this.  They store the lastPollTimestamp internally and compare that to
>> the file's timestamps to determine if a file still needs to be
>> processed.  However, the lastPollTime isn't being persisted anywhere, so
>> restarting the JVM will result in re-reading the files.
>>
>> If you want, you can always provide a patch for the file-handling
>> strategies you want to use for your project.  Let us know if you need
>> any help with that...  Any (even partial) solution for this issue would
>> be more than welcome.
>>
>> If you are using Camel inside ServiceMix, you can also use ServiceMix's
>> FTP poller component as a (temporary) workaround.  That one does already
>> support most forms of file-handling (delete, move, ...).
>>
>>
>> Regards,
>>
>> Gert
>>
>>
>>
>> Drew McAuliffe wrote:
>>> Am I crazy or is there no way that the FTP/SFTP consumers do anything to
>>> a
>>> source file once they've read it? There doesn't appear to be any delete
>>> or
>>> move option like with the File consumer. What possible use case would
>>> that
>>> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer
>>> will
>>> continuously re-read files, never stopping?
>>>
>>> It appears that there's a JIRA issue to add support for something
>>> similar
>>> to
>>> a file renaming strategy, but it doesn't look like it's slated for
>>> anything
>>> in 1.4.
>>>  
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17684963.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17700718.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Drew McAuliffe
Claus,

I uploaded a zip with 3 files changed to CAMEL-570. Let me know if there are any issues.

Drew
Claus Ibsen wrote
Hi Drew

Wrapping into a runtime exception is okay, but try to find a suitable exception in camel-core.

I do think that Camel could be improved here to adhere to a more common/strict exception hieracy - but that can be a goal for Camel 2.0 where the API can be changed slightly.

The FTPConsumer will have it's exception handled by the ScheduledPollConsumer#run.


About the async, well I do not think so at current time. Is there any real
life use-case for this requirements?

Let's keep the task simple to remedy the problem with the lack of a delete=true|false option on the component.


Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Drew McAuliffe [mailto:drewmca@gmail.com]
Sent: 6. juni 2008 22:39
To: camel-user@activemq.apache.org
Subject: RE: No way to remove FTP/SFTP files?


Before I submit it, any ideas on how to handle the exception throwing in the
processing? I'm just throwing a RuntimeException because I didn't know what
the normal Camel procedure was. I have to trap the IOException to adhere to
the interface of the async processor.

Also, are there any repercussions for doing this in an async processor
block, as opposed to the way the component handles it now (synchronously
with a call directly to "process")?



Claus Ibsen wrote:
>
> Hi Drew
>
> Great work. Keep it up and I will later commit it into Camel.
> When you think its ready then please submit it as a patch to CAMEL-570.
>
> Camel does really need this feature to remove consumed FTP files.
> Transferring messages using FTP is still a very common use case in real
> life.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
>
> -----Original Message-----
> From: Drew McAuliffe [mailto:drewmca@gmail.com]
> Sent: 6. juni 2008 07:35
> To: camel-user@activemq.apache.org
> Subject: RE: No way to remove FTP/SFTP files?
>
>
> Just as an FYI, I've hacked the following (after adding an "autoDelete"
> property to the endpoint configuration. This is in "pollFile" in
> FtpConsumer. It's a little staggered for debugging purposes, and I'm sure
> the exception throwing isn't ideal, but it seems to work. This is based
> somewhat on the Mule FTP and SFTP components. I'm working on the SFTP
> version next. While I think a FileProcessStrategy approach similar to the
> File component might be nice, for now an autodelete option works just fine
> (I delete the files read from SFTP after moving them to local storage,
> where
> I can handle them with proper backup after reading). Again, I think if
> you're going to go through the effort of implementing some sort of
> FileProcessStrategy thing, it would be better to do it once using a VFS
> component instead of splitting it between FTP and SFTP because of the
> different clients.
>
>                 getAsyncProcessor().process(exchange, new AsyncCallback(){
>                 public void done(boolean sync) {
>                         boolean failed = exchange.isFailed();
>                         boolean handled =
> DeadLetterChannel.isFailureHandled(exchange);
>                        
>                         if (LOG.isDebugEnabled()) {
>                             LOG.debug("Done processing file: " + ftpFile +
> ". Status is: " + (failed ? "failed: " + failed + ", handled by failure
> processor: " + handled : "OK"));
>                         }
>
>                         if (!failed || handled) {
>                         // if autodelete, then remove the source file
>                         if (endpoint.getConfiguration().isAutoDelete()){
>                         try {
>                         String toDelete = getFullFileName(ftpFile);
> boolean deleted = client.deleteFile(toDelete);
> if (! deleted)
> throw new IOException(MessageFormat.format("Could not delete
> remote file at path {0}. Ftp error:{1}",
> new Object[]{ftpFile.getName(), client.getReplyCode()}));
>
> } catch (IOException e) {
> throw new RuntimeException(e.getMessage(), e);
> }
>                         }
>                         } else if (failed && !handled) {
>                             // there was an exception but it was not
> handled
> by the DeadLetterChannel
>                             handleException(exchange.getException());
>                         }
>                 }
>                 });
>  
>
>
> Claus Ibsen wrote:
>>
>> Hi
>>
>> Added ticket: CAMEL-570 to track this issue.
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. +45 2962 7576
>> Web: www.silverbullet.dk
>>
>> -----Original Message-----
>> From: Claus Ibsen [mailto:ci@silverbullet.dk]
>> Sent: 3. juni 2008 05:50
>> To: camel-user@activemq.apache.org
>> Subject: RE: No way to remove FTP/SFTP files?
>>
>> Hi
>>
>> And voting for the tickets also helps the team decide which issues we
>> should tackle first.
>>
>> Personally I do think file based components needs an overhaul in Camel to
>> improve their standards as this is IMHO still one of the most common
>> integration techniques in my daily fields of work.
>>
>> Gert good point about that ServiceMix FTP component. Good spot for code
>> resuse.
>>
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. +45 2962 7576
>> Web: www.silverbullet.dk
>> -----Original Message-----
>> From: Gert Vanthienen [mailto:gert.vanthienen@skynet.be]
>> Sent: 3. juni 2008 04:42
>> To: camel-user@activemq.apache.org
>> Subject: Re: No way to remove FTP/SFTP files?
>>
>> Drew,
>>
>>
>> No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
>> this.  They store the lastPollTimestamp internally and compare that to
>> the file's timestamps to determine if a file still needs to be
>> processed.  However, the lastPollTime isn't being persisted anywhere, so
>> restarting the JVM will result in re-reading the files.
>>
>> If you want, you can always provide a patch for the file-handling
>> strategies you want to use for your project.  Let us know if you need
>> any help with that...  Any (even partial) solution for this issue would
>> be more than welcome.
>>
>> If you are using Camel inside ServiceMix, you can also use ServiceMix's
>> FTP poller component as a (temporary) workaround.  That one does already
>> support most forms of file-handling (delete, move, ...).
>>
>>
>> Regards,
>>
>> Gert
>>
>>
>>
>> Drew McAuliffe wrote:
>>> Am I crazy or is there no way that the FTP/SFTP consumers do anything to
>>> a
>>> source file once they've read it? There doesn't appear to be any delete
>>> or
>>> move option like with the File consumer. What possible use case would
>>> that
>>> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer
>>> will
>>> continuously re-read files, never stopping?
>>>
>>> It appears that there's a JIRA issue to add support for something
>>> similar
>>> to
>>> a file renaming strategy, but it doesn't look like it's slated for
>>> anything
>>> in 1.4.
>>>  
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17684963.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17700718.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

RE: No way to remove FTP/SFTP files?

Claus Ibsen
Hi Drew

Thanks for the hard work and observations on the FTP/SFTP components.
We really appreicate contributions from the community - thanks.

As Camel 1.4 is just about to get released, we will target your patches for 1.5.

Med venlig hilsen
 
Claus Ibsen
......................................
Silverbullet
Skovsgårdsvænget 21
8362 Hørning
Tlf. +45 2962 7576
Web: www.silverbullet.dk
-----Original Message-----
From: Drew McAuliffe [mailto:[hidden email]]
Sent: 28. juni 2008 03:37
To: [hidden email]
Subject: RE: No way to remove FTP/SFTP files?


Claus,

I uploaded a zip with 3 files changed to CAMEL-570. Let me know if there are
any issues.

Drew

Claus Ibsen wrote:

>
> Hi Drew
>
> Wrapping into a runtime exception is okay, but try to find a suitable
> exception in camel-core.
>
> I do think that Camel could be improved here to adhere to a more
> common/strict exception hieracy - but that can be a goal for Camel 2.0
> where the API can be changed slightly.
>
> The FTPConsumer will have it's exception handled by the
> ScheduledPollConsumer#run.
>
>
> About the async, well I do not think so at current time. Is there any real
> life use-case for this requirements?
>
> Let's keep the task simple to remedy the problem with the lack of a
> delete=true|false option on the component.
>
>
> Med venlig hilsen
>  
> Claus Ibsen
> ......................................
> Silverbullet
> Skovsgårdsvænget 21
> 8362 Hørning
> Tlf. +45 2962 7576
> Web: www.silverbullet.dk
> -----Original Message-----
> From: Drew McAuliffe [mailto:[hidden email]]
> Sent: 6. juni 2008 22:39
> To: [hidden email]
> Subject: RE: No way to remove FTP/SFTP files?
>
>
> Before I submit it, any ideas on how to handle the exception throwing in
> the
> processing? I'm just throwing a RuntimeException because I didn't know
> what
> the normal Camel procedure was. I have to trap the IOException to adhere
> to
> the interface of the async processor.
>
> Also, are there any repercussions for doing this in an async processor
> block, as opposed to the way the component handles it now (synchronously
> with a call directly to "process")?
>
>
>
> Claus Ibsen wrote:
>>
>> Hi Drew
>>
>> Great work. Keep it up and I will later commit it into Camel.
>> When you think its ready then please submit it as a patch to CAMEL-570.
>>
>> Camel does really need this feature to remove consumed FTP files.
>> Transferring messages using FTP is still a very common use case in real
>> life.
>>
>>
>> Med venlig hilsen
>>  
>> Claus Ibsen
>> ......................................
>> Silverbullet
>> Skovsgårdsvænget 21
>> 8362 Hørning
>> Tlf. +45 2962 7576
>> Web: www.silverbullet.dk
>>
>> -----Original Message-----
>> From: Drew McAuliffe [mailto:[hidden email]]
>> Sent: 6. juni 2008 07:35
>> To: [hidden email]
>> Subject: RE: No way to remove FTP/SFTP files?
>>
>>
>> Just as an FYI, I've hacked the following (after adding an "autoDelete"
>> property to the endpoint configuration. This is in "pollFile" in
>> FtpConsumer. It's a little staggered for debugging purposes, and I'm sure
>> the exception throwing isn't ideal, but it seems to work. This is based
>> somewhat on the Mule FTP and SFTP components. I'm working on the SFTP
>> version next. While I think a FileProcessStrategy approach similar to the
>> File component might be nice, for now an autodelete option works just
>> fine
>> (I delete the files read from SFTP after moving them to local storage,
>> where
>> I can handle them with proper backup after reading). Again, I think if
>> you're going to go through the effort of implementing some sort of
>> FileProcessStrategy thing, it would be better to do it once using a VFS
>> component instead of splitting it between FTP and SFTP because of the
>> different clients.
>>
>>                 getAsyncProcessor().process(exchange, new
>> AsyncCallback(){
>>                 public void done(boolean sync) {
>>                         boolean failed = exchange.isFailed();
>>                         boolean handled =
>> DeadLetterChannel.isFailureHandled(exchange);
>>                        
>>                         if (LOG.isDebugEnabled()) {
>>                             LOG.debug("Done processing file: " + ftpFile
>> +
>> ". Status is: " + (failed ? "failed: " + failed + ", handled by failure
>> processor: " + handled : "OK"));
>>                         }
>>
>>                         if (!failed || handled) {
>>                         // if autodelete, then remove the source file
>>                         if (endpoint.getConfiguration().isAutoDelete()){
>>                         try {
>>                         String toDelete = getFullFileName(ftpFile);
>> boolean deleted = client.deleteFile(toDelete);
>> if (! deleted)
>> throw new IOException(MessageFormat.format("Could not delete
>> remote file at path {0}. Ftp error:{1}",
>> new Object[]{ftpFile.getName(), client.getReplyCode()}));
>>
>> } catch (IOException e) {
>> throw new RuntimeException(e.getMessage(), e);
>> }
>>                         }
>>                         } else if (failed && !handled) {
>>                             // there was an exception but it was not
>> handled
>> by the DeadLetterChannel
>>                             handleException(exchange.getException());
>>                         }
>>                 }
>>                 });
>>  
>>
>>
>> Claus Ibsen wrote:
>>>
>>> Hi
>>>
>>> Added ticket: CAMEL-570 to track this issue.
>>>
>>>
>>> Med venlig hilsen
>>>  
>>> Claus Ibsen
>>> ......................................
>>> Silverbullet
>>> Skovsgårdsvænget 21
>>> 8362 Hørning
>>> Tlf. +45 2962 7576
>>> Web: www.silverbullet.dk
>>>
>>> -----Original Message-----
>>> From: Claus Ibsen [mailto:[hidden email]]
>>> Sent: 3. juni 2008 05:50
>>> To: [hidden email]
>>> Subject: RE: No way to remove FTP/SFTP files?
>>>
>>> Hi
>>>
>>> And voting for the tickets also helps the team decide which issues we
>>> should tackle first.
>>>
>>> Personally I do think file based components needs an overhaul in Camel
>>> to
>>> improve their standards as this is IMHO still one of the most common
>>> integration techniques in my daily fields of work.
>>>
>>> Gert good point about that ServiceMix FTP component. Good spot for code
>>> resuse.
>>>
>>>
>>>
>>> Med venlig hilsen
>>>  
>>> Claus Ibsen
>>> ......................................
>>> Silverbullet
>>> Skovsgårdsvænget 21
>>> 8362 Hørning
>>> Tlf. +45 2962 7576
>>> Web: www.silverbullet.dk
>>> -----Original Message-----
>>> From: Gert Vanthienen [mailto:[hidden email]]
>>> Sent: 3. juni 2008 04:42
>>> To: [hidden email]
>>> Subject: Re: No way to remove FTP/SFTP files?
>>>
>>> Drew,
>>>
>>>
>>> No, you are not crazy.  Currently, the FTP/SFTP consumers don't support
>>> this.  They store the lastPollTimestamp internally and compare that to
>>> the file's timestamps to determine if a file still needs to be
>>> processed.  However, the lastPollTime isn't being persisted anywhere, so
>>> restarting the JVM will result in re-reading the files.
>>>
>>> If you want, you can always provide a patch for the file-handling
>>> strategies you want to use for your project.  Let us know if you need
>>> any help with that...  Any (even partial) solution for this issue would
>>> be more than welcome.
>>>
>>> If you are using Camel inside ServiceMix, you can also use ServiceMix's
>>> FTP poller component as a (temporary) workaround.  That one does already
>>> support most forms of file-handling (delete, move, ...).
>>>
>>>
>>> Regards,
>>>
>>> Gert
>>>
>>>
>>>
>>> Drew McAuliffe wrote:
>>>> Am I crazy or is there no way that the FTP/SFTP consumers do anything
>>>> to
>>>> a
>>>> source file once they've read it? There doesn't appear to be any delete
>>>> or
>>>> move option like with the File consumer. What possible use case would
>>>> that
>>>> support? Doesn't this pretty much guarantee that an FTP/SFTP consumer
>>>> will
>>>> continuously re-read files, never stopping?
>>>>
>>>> It appears that there's a JIRA issue to add support for something
>>>> similar
>>>> to
>>>> a file renaming strategy, but it doesn't look like it's slated for
>>>> anything
>>>> in 1.4.
>>>>  
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17684963.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>>
>>
>>
>
> --
> View this message in context:
> http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p17700718.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>

--
View this message in context: http://www.nabble.com/No-way-to-remove-FTP-SFTP-files--tp17612896s22882p18166504.html
Sent from the Camel - Users mailing list archive at Nabble.com.