Quantcast

Camel-FTP component on cluster

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Camel-FTP component on cluster

yogu13
Hi,

I would like to know if the camel-FTP component supports cluster? we have a requirement of downloading files kept at FTP location and processing these files once downloaded. List of Software's which are used in the application:-

Camel - version 2.10.1
JDK - version 1.6.37
Server - Tomcat 7

Our Camel Application is bundled into a WAR and deployed on to tomcat and expected to run on cluster in production. According to my understanding of the component the camel-ftp endpoint would be running on both the nodes in tomcat cluster and would download the same files simultaneously and start processing, which is not what we want. We would like only one of the nodes to download the file and process it accordinly...

Please clarify if my understanding is right and also let me know options that we have to get this functionality done using camel.

I am also thinking on the lines of using Quartz and having it to poll ever 10 seconds to FTP location and download the file if found and process it as Quartz supports clustering but with this option i think we will have to write a custom processor code for checking the ftp location and downloading instead of reusing the camel-ftp endpoint.

Thanks in advance...

Cheers!
-Yogesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel-FTP component on cluster

Claus Ibsen-2
Hi

This has been debated before about cluster Camel. Maybe you can search
the forum and find some of these talks.

You can use a route policy to control which route is master / slave etc
http://camel.apache.org/routepolicy

And use some different technologies for that such as quartz,
hazelcast, zookeeper, et all.
Or write some custom code etc.

And in Fuse ESB Enterprise there is a Camel master component out of
the box that does this for you.

Though an alternative is to use a shared idempotent repository that
the ftp consumer can use.
That is a bit special as the file/ftp consumer has this functionality
built-in. Not all Camel consumers offers that.
And hence why a general solution with the route policy can be used.



On Tue, Mar 19, 2013 at 10:03 AM, yogu13 <[hidden email]> wrote:

> Hi,
>
> I would like to know if the camel-FTP component supports cluster? we have a
> requirement of downloading files kept at FTP location and processing these
> files once downloaded. List of Software's which are used in the
> application:-
>
> Camel - version 2.10.1
> JDK - version 1.6.37
> Server - Tomcat 7
>
> Our Camel Application is bundled into a WAR and deployed on to tomcat and
> expected to run on cluster in production. According to my understanding of
> the component the camel-ftp endpoint would be running on both the nodes in
> tomcat cluster and would download the same files simultaneously and start
> processing, which is not what we want. We would like only one of the nodes
> to download the file and process it accordinly...
>
> Please clarify if my understanding is right and also let me know options
> that we have to get this functionality done using camel.
>
> I am also thinking on the lines of using Quartz and having it to poll ever
> 10 seconds to FTP location and download the file if found and process it as
> Quartz supports clustering but with this option i think we will have to
> write a custom processor code for checking the ftp location and downloading
> instead of reusing the camel-ftp endpoint.
>
> Thanks in advance...
>
> Cheers!
> -Yogesh
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-component-on-cluster-tp5729403.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel-FTP component on cluster

yogu13
Hi Claus,

Can you let us know what do you mean by "Though an alternative is to use a shared idempotent repository that the ftp consumer can use. That is a bit special as the file/ftp consumer has this functionality built-in"

As mentioned will try to search the discussion group as well on clustered camel.

Thank you!

Regards,
-Yogesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel-FTP component on cluster

Claus Ibsen-2
On Tue, Mar 19, 2013 at 12:34 PM, yogu13 <[hidden email]> wrote:
> Hi Claus,
>
> Can you let us know what do you mean by "Though an alternative is to use a
> shared idempotent repository that the ftp consumer can use. That is a bit
> special as the file/ftp consumer has this functionality built-in"
>

Yeah this allows you to have all nodes in the cluster be active and
try to pickup FTP files.
And with a shared idempotent repository configured on them, this can
ensure a file
is only being picked up by a single node.


Where as the "Camel cluster" using quartz, hazelcast etc. is using a
strategy of one node
being a master, and the others slaves. And when the master dies, one
of the slaves become the new master etc.
But then there is only one active node running.






> As mentioned will try to search the discussion group as well on clustered
> camel.
>
> Thank you!
>
> Regards,
> -Yogesh
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-component-on-cluster-tp5729403p5729407.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel-FTP component on cluster

yogu13
Hi Claus,

Looks like i finally would we working on getting this component up on the cluster...Hence re initiating the thread with you.. wanted to know, How can i configure a shared idempotent repository for a ftp component. My apologies i do not see any such option with the configurations that the ftp component provides.

Thank you!

Regards,
-Yogesh
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Camel-FTP component on cluster

Claus Ibsen-2
See the idempotentRepository option
http://camel.apache.org/file2

On Fri, Jun 21, 2013 at 12:31 PM, yogu13 <[hidden email]> wrote:

> Hi Claus,
>
> Looks like i finally would we working on getting this component up on the
> cluster...Hence re initiating the thread with you.. wanted to know, How can
> i configure a shared idempotent repository for a ftp component. My apologies
> i do not see any such option with the configurations that the ftp component
> provides.
>
> Thank you!
>
> Regards,
> -Yogesh
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-component-on-cluster-tp5729403p5734518.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
www.camelone.org: The open source integration conference.

Red Hat, Inc.
FuseSource is now part of Red Hat
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RE: Camel-FTP component on cluster

siano
In reply to this post by yogu13
Hi,
Even though I am not Claus, you need to set up a shared implementation of the IdempotentRepository. The ones from Camel-Core (File and Memory) are not shared, but you could use the implementation from the SQL, JPA or Hazelcast component (or write your own).

In case of a cluster you may also need a inProgressRepository (which is another idempotent repository) that is shared between cluster nodes.

Best regards
Stephan

-----Original Message-----
From: yogu13 [mailto:[hidden email]]
Sent: Freitag, 21. Juni 2013 12:32
To: [hidden email]
Subject: Re: Camel-FTP component on cluster

Hi Claus,

Looks like i finally would we working on getting this component up on the
cluster...Hence re initiating the thread with you.. wanted to know, How can
i configure a shared idempotent repository for a ftp component. My apologies
i do not see any such option with the configurations that the ftp component
provides.

Thank you!

Regards,
-Yogesh



--
View this message in context: http://camel.465427.n5.nabble.com/Camel-FTP-component-on-cluster-tp5729403p5734518.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Loading...