Quantcast

Camel FTP/SFTP check instead of download

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Camel FTP/SFTP check instead of download

Nestor Urquiza
Hello riders,

There are times when all you want is to check for the existence of a file, or the timestamp of it however that is not apparently available in the FTP/SFTP component as per http://camel.apache.org/ftp.html.

Does it make sense to have a feature request to allow for this kind of check?

Thanks,
- Nestor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Babak Vahdat
Hi

For Camel 2.x the documentation is at:

http://camel.apache.org/ftp2.html

and not:

http://camel.apache.org/ftp.html

And for the purpose you mentioned, using the filtering support should easily do the trick:

http://camel.apache.org/ftp2.html#FTP2-Filterusing%7B%7Borg.apache.camel.component.file.GenericFileFilter%7D%7D

Babak
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Nestor Urquiza
Thanks Babak,

I am trying to activate a process chain only if certain file in on the remote sftp. Probably you or another reader can help with some errors I am getting.

If I use wrong credentials Camel tries to reconnect several times not obeying to maximumReconnectAttempts=0:
<code>
.from("sftp://sample.com/home/user/test?username=user&password=wrongPass&fastExistsCheck=true&maximumReconnectAttempts=0")      
</code

If I use good credentials and a typical home path like above it complaints "Cannot change directory
to: home" so I am forced to use a double slash
<code>
.from("sftp://sample.com//home/user/test?username=user&password=pass&fastExistsCheck=true")      
</code

But then with the above I get:
<quote>
Can't change directory: /home/krfsadmin/test


If I use option "directory=false" which is not in the ftp2 documentation but in the ftp dcumentation: then I get:

Cannot change directory to: false


Best,
-Nestor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Claus Ibsen-2
On Wed, May 23, 2012 at 9:56 PM, Nestor Urquiza
<[hidden email]> wrote:

> Thanks Babak,
>
> I am trying to activate a process chain only if certain file in on the
> remote sftp. Probably you or another reader can help with some errors I am
> getting.
>
> If I use wrong credentials Camel tries to reconnect several times not
> obeying to maximumReconnectAttempts=0:
> <code>
> .from("sftp://sample.com/home/user/test?username=user&password=wrongPass&fastExistsCheck=true&maximumReconnectAttempts=0")
> </code
>

Do you see 3 login attempts or what?
What version of Camel do you use?



> If I use good credentials and a typical home path like above it complaints
> &quot;Cannot change directory
> to: home&quot; so I am forced to use a double slash
> &lt;code>
> .from("sftp://sample.com//home/user/test?username=user&password=pass&fastExistsCheck=true")
> </code
>
> But then with the above I get:
> <quote>
> Can't change directory: /home/krfsadmin/test
>

Maybe dont have a leading slash. It's really about the FTP server in
the other end, not allow you to change to that dir. Check on that end.
For example if the FTP server already have a home dir, then you don't
need to specify a starting directory.

 .from("sftp://sample.com?username=user&password=pass&fastExistsCheck=true")



>
> If I use option "directory=false" which is not in the ftp2 documentation but
> in the ftp dcumentation: then I get:
>

As already said before. the old documentation at ftp is for Camel 1.x.
So there is no such option in Camel 2.x.

There is a fileName option, if you want to filter on a specific file
name only, etc.
Please spend a bit more time reading the docs and try out.


> Cannot change directory to: false
>
>
> Best,
> -Nestor
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-SFTP-check-instead-of-download-tp5713363p5713432.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Claus Ibsen-2
In reply to this post by Nestor Urquiza
On Wed, May 23, 2012 at 2:47 AM, Nestor Urquiza
<[hidden email]> wrote:
> Hello riders,
>
> There are times when all you want is to check for the existence of a file,
> or the timestamp of it however that is not apparently available in the
> FTP/SFTP component as per http://camel.apache.org/ftp.html.
>
> Does it make sense to have a feature request to allow for this kind of
> check?
>

No not on the consumer.

There is an EIP for that
http://camel.apache.org/polling-consumer

You can then use receive no wait to try to poll from an endpoint,
where you specify the filename you want to check.

Also you can use the camel-ftp component API to get hold on the remote
file operations API, and invoke the methods there.
There is methods to list directory etc. Then you can check from java code, etc.


> Thanks,
> - Nestor
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-SFTP-check-instead-of-download-tp5713363.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
CamelOne 2012 Conference, May 15-16, 2012: http://camelone.com
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.blogspot.com/
Author of Camel in Action: http://www.manning.com/ibsen/
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Nestor Urquiza
Thanks Clous. Your answers brought me back to earth.

I think I missed the point about what the ftp components and any other Camel component is about (Producer/Consumer). I was trying to activate a route from a seda component and then only at that point try to test for the existence of a file in the remote SFTP. If that is what I need I should use just a bean and not really the component. If the need is to constantly check for the file existence then we just use the Polling Consumer pattern.

My needs (send an email when there is a new file) could be covered (tested and it works) with a filter like below:
<code>
        public class FileAttributesFilter<T> implements GenericFileFilter<T> {

        public boolean accept(GenericFile<T> file) {
            return file.getLastModified() + RECONNECT_DELAY_MSEC > System.currentTimeMillis();
        }
    }
</code>

However the next question is how to make sure two JVMs running the same Camel routes do not compete with each other. If I use a persisted quartz route and a bean I can be sure that only one one JVM will run the quartz StatefulJob which can produce the message for a unique end point in that VM. How can I achieve the same with the Camel FTP Consumer?

Also it looks like Camel FTP pulls the files even when using fastExistsCheck=true, is that expected? I know it does pull the file because if I chain an smtp component I get the content of the file by email.

On my initial tests (for the record) there were several miss understanding generating all errors I posted:
1. As it implements the Polling Consumer Pattern Camel FTP will continue to poll, you can control the delay and other parameters but that won't mean it will stop retrying even if the problem is related to bad credentials.
2. After providing a username any path after the domain is relative to the home directory so should you need absolute paths you need to add a slash so "sftp://sample.com//home/admin?username=admin" is really equivalent to just "sftp://sample.com?username=admin" (most of the time)
3. FTP2 inherits all fields from File2 between which there is an option to pull just one file. Implementing a GenericFileFilter is an alternative that goes beyond that as you can literally check for any file attributes before accepting the file as a valid message for the route.

Best,

-Nestor
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Claus Ibsen-2
On Fri, May 25, 2012 at 7:52 PM, Nestor Urquiza
<[hidden email]> wrote:

> Thanks Clous. Your answers brought me back to earth.
>
> I think I missed the point about what the ftp components and any other Camel
> component is about (Producer/Consumer). I was trying to activate a route
> from a seda component and then only at that point try to test for the
> existence of a file in the remote SFTP. If that is what I need I should use
> just a bean and not really the component. If the need is to constantly check
> for the file existence then we just use the Polling Consumer pattern.
>
> My needs (send an email when there is a new file) could be covered (tested
> and it works) with a filter like below:
> <code>
>        public class FileAttributesFilter<T> implements GenericFileFilter<T> {
>
>        public boolean accept(GenericFile<T> file) {
>            return file.getLastModified() + RECONNECT_DELAY_MSEC >
> System.currentTimeMillis();
>        }
>    }
> </code>
>
> However the next question is how to make sure two JVMs running the same
> Camel routes do not compete with each other. If I use a persisted quartz
> route and a bean I can be sure that only one one JVM will run the quartz
> StatefulJob which can produce the message for a unique end point in that VM.
> How can I achieve the same with the Camel FTP Consumer?
>

You can for example use a route policy and figure out the locking
strategy yourself.
http://camel.apache.org/routepolicy

Maybe tie that into quartz clustering support somehow.


The Fuse Fabric has a Camel master component that does a master/slave
for Camel routes in a cluster.
https://github.com/fusesource/fuse/blob/master/fabric/fabric-camel/src/main/java/org/fusesource/fabric/camel/MasterComponent.java

Does not updated yet
http://fuse.fusesource.org/fabric/docs/user-guide.html#Camel_Fabric

That requires to use Fabric though, in eg Apache Karaf / ServiceMix or
the Fuse products/


> Also it looks like Camel FTP pulls the files even when using
> fastExistsCheck=true, is that expected? I know it does pull the file because
> if I chain an smtp component I get the content of the file by email.
>

Yeah it always download the file. We could possible add option
download=true|false, to the ftp component.
Then ppl can use download=false, to be notified of the existence of new files.
Fell free to create a JIRA for that idea.



> On my initial tests (for the record) there were several miss understanding
> generating all errors I posted:
> 1. As it implements the Polling Consumer Pattern Camel FTP will continue to
> poll, you can control the delay and other parameters but that won't mean it
> will stop retrying even if the problem is related to bad credentials.
> 2. After providing a username any path after the domain is relative to the
> home directory so should you need absolute paths you need to add a slash so
> "sftp://sample.com//home/admin?username=admin" is really equivalent to just
> "sftp://sample.com?username=admin" (most of the time)
> 3. FTP2 inherits all fields from File2 between which there is an option to
> pull just one file. Implementing a GenericFileFilter is an alternative that
> goes beyond that as you can literally check for any file attributes before
> accepting the file as a valid message for the route.
>
> Best,
>
> -Nestor
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-SFTP-check-instead-of-download-tp5713363p5713567.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Nestor Urquiza
Hi Claus,

I just created https://issues.apache.org/jira/browse/CAMEL-5320

Thanks again for your suggestions. I think for now I found how to rely on the clustering capabilities of my current tomcat sticky session deployments for load balancing. If the process is initiated from a regular http request or scheduled (quartz) I can trigger the routes which for now can run in just one JVM once they start. If I need more distributed load I will need to jump into inter JVM messaging (AMQP sounds good to me).

BTW I have blogged about my solution to use Camel as Service Orchestrator in http://thinkinginsoftware.blogspot.com/2012/05/using-quartz-camel-and-spring-for.html, hopefully my journey will help others looking for simple solutions to big problems.

Camel rocks!

-Nestor

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel FTP/SFTP check instead of download

Claus Ibsen-2
On Wed, May 30, 2012 at 6:12 PM, Nestor Urquiza
<[hidden email]> wrote:

> Hi Claus,
>
> I just created https://issues.apache.org/jira/browse/CAMEL-5320
>
> Thanks again for your suggestions. I think for now I found how to rely on
> the clustering capabilities of my current tomcat sticky session deployments
> for load balancing. If the process is initiated from a regular http request
> or scheduled (quartz) I can trigger the routes which for now can run in just
> one JVM once they start. If I need more distributed load I will need to jump
> into inter JVM messaging (AMQP sounds good to me).
>
> BTW I have blogged about my solution to use Camel as Service Orchestrator in
> http://thinkinginsoftware.blogspot.com/2012/05/using-quartz-camel-and-spring-for.html,
> hopefully my journey will help others looking for simple solutions to big
> problems.
>

Ah a nice and detailed blog. I took the liberty to add a link to it
from the Camel articles page
(takes a bit to update)
http://camel.apache.org/articles


> Camel rocks!
>
> -Nestor
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-SFTP-check-instead-of-download-tp5713363p5713744.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Loading...