distributing files among clusters

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

distributing files among clusters

ayrshid99
Hi All,

i have few camel routes which are reading from a directory say Samples. the
camel is running on multiple nodes and all systems are reading files from
centralized location.
my question how can i distribute files equally among all nodes.


--
Arshid
Reply | Threaded
Open this post in threaded view
|

Re: distributing files among clusters

Farkas Lajos-2
Hello,

the simplest way

1.create one route which gets all the files or their names,  and puts them
on  a queue (activemq activemq)
2. the processing routes from the clusters are all listening  on the queue,
and by the way the queue works, each of them will receive different
filenames to process, in a distributed way

Regards
Laji



On Fri, Aug 2, 2019 at 1:02 PM arshad dar <[hidden email]> wrote:

> Hi All,
>
> i have few camel routes which are reading from a directory say Samples. the
> camel is running on multiple nodes and all systems are reading files from
> centralized location.
> my question how can i distribute files equally among all nodes.
>
>
> --
> Arshid
>
Reply | Threaded
Open this post in threaded view
|

Re: distributing files among clusters

ayrshid99
Thanks for your response.I have around 50 routes and each routes is
replicated on 5 nodes.The way you suggested isn't it possible that all the
50 files will be processed by one node while other 4 nodes will remain
idle.

On Wed, 7 Aug, 2019, 4:46 AM Farkas Lajos, <[hidden email]> wrote:

> Hello,
>
> the simplest way
>
> 1.create one route which gets all the files or their names,  and puts them
> on  a queue (activemq activemq)
> 2. the processing routes from the clusters are all listening  on the queue,
> and by the way the queue works, each of them will receive different
> filenames to process, in a distributed way
>
> Regards
> Laji
>
>
>
> On Fri, Aug 2, 2019 at 1:02 PM arshad dar <[hidden email]> wrote:
>
> > Hi All,
> >
> > i have few camel routes which are reading from a directory say Samples.
> the
> > camel is running on multiple nodes and all systems are reading files from
> > centralized location.
> > my question how can i distribute files equally among all nodes.
> >
> >
> > --
> > Arshid
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: distributing files among clusters

Claus Ibsen-2
In reply to this post by ayrshid99
Hi

If you have a copy of the Camel in Action 2nd edition book, then the
concurrency chapter covers how to distribute and process files
concurrently.


On Fri, Aug 2, 2019 at 12:02 PM arshad dar <[hidden email]> wrote:

>
> Hi All,
>
> i have few camel routes which are reading from a directory say Samples. the
> camel is running on multiple nodes and all systems are reading files from
> centralized location.
> my question how can i distribute files equally among all nodes.
>
>
> --
> Arshid



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: distributing files among clusters

Farkas Lajos-2
In reply to this post by ayrshid99
If you use queues, messages will be distributed among the listeners on the
queue, it is the way the messaging queue works. All you have to make sure
is that the processing processes are listening on the same queue. We use
this to distribute easily load between multiple processing nodes.


LAji

On Thu, Aug 8, 2019 at 8:21 AM arshad dar <[hidden email]> wrote:

> Thanks for your response.I have around 50 routes and each routes is
> replicated on 5 nodes.The way you suggested isn't it possible that all the
> 50 files will be processed by one node while other 4 nodes will remain
> idle.
>
> On Wed, 7 Aug, 2019, 4:46 AM Farkas Lajos, <[hidden email]>
> wrote:
>
> > Hello,
> >
> > the simplest way
> >
> > 1.create one route which gets all the files or their names,  and puts
> them
> > on  a queue (activemq activemq)
> > 2. the processing routes from the clusters are all listening  on the
> queue,
> > and by the way the queue works, each of them will receive different
> > filenames to process, in a distributed way
> >
> > Regards
> > Laji
> >
> >
> >
> > On Fri, Aug 2, 2019 at 1:02 PM arshad dar <[hidden email]> wrote:
> >
> > > Hi All,
> > >
> > > i have few camel routes which are reading from a directory say Samples.
> > the
> > > camel is running on multiple nodes and all systems are reading files
> from
> > > centralized location.
> > > my question how can i distribute files equally among all nodes.
> > >
> > >
> > > --
> > > Arshid
> > >
> >
>