Using correlationid to isolate jms queues in multiple camelContexts

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Using correlationid to isolate jms queues in multiple camelContexts

garrydias
Dear friends

I´m facing the folowing problem: I have 2 producers systems sending messages to 1 jms queue that is consumed by 2 consumers systems. Ignore the VM where they are.

#a) a producerSystem_1.war has the folowing camelContext:

<camelContext id="producerSystem-1-camelContext">
  ...
  <route>
    <from uri="direct:whatever" />
    <to uri="activemq:common_queue" />
  </route>
  ...
</camelContext >


#b) a producerSystem_2.war has the folowing camelContext:

<camelContext id="producerSystem-2-camelContext">
  ...
  <route>
    <from uri="direct:whatever" />
    <to uri="activemq:common_queue" />
  </route>
  ...
</camelContext >


#c) a consumerSystem_1.war has the folowing camelContext:

<camelContext id="context_1">
  <route>
    <from uri="activemq:common_queue" />
    <to uri="bean:consumer_a" />
  </route>
</camelContext >


#d) a consumerSystem_2.war has the folowing camelContext:

<camelContext id="context_2">
  <route>
    <from uri="activemq:common_queue" />
    <to uri="bean:consumer_b" />
  </route>
</camelContext >


The VM where these applications are running is not important.

My desired behavior is:
- Messages produced in #a must consumed in c#
- Messages produced in #b must consumed in d#

The actual behavior is:
- Messages produced in #a sometimes is consumed in c# and sometimes in d#
- Messages produced in #b sometimes is consumed in c# and sometimes in d#

I understand that JMS endpoints were designed to execute in the actual behavior. But in this case, I need to run my desired behavior.

There´s a way to do this using correlationIds (not send/receive approach)?

Thanx a lot
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

pontus.ullgren
Hello,

The way you see it behave is the way it should behave. If you have
competing consumers on a queue they will consume messages in that way.

If I interpreted your requirement correctly you have messages from a#
that can only be consumed by c# and messages from b# that can only be
consumed by d#.
In that case you should really use two different queues for this one
between each pair of producer and consumer.

Adding extra queues in activemq is simple and very low cost when it
comes to resources. So I would advice you to solve it this way.

If you for some reason still want to use one queue (legacy, crazy
customer requirement or let's say that you are building a example for
bad architecture ;-P) then you can set a header on the messages in the
producer and then use a selector in the consumer. See the
documentation on the Camel JMS component[1] and the JMS component unit
test.

But again: Use two queues.

[1] http://camel.apache.org/jms.html
[2] https://svn.apache.org/repos/asf/camel/trunk/components/camel-jms/src/test/java/org/apache/camel/component/jms/JmsSelectorTest.java
Best regards
Pontus Ullgren


On Wed, Sep 12, 2012 at 7:18 PM, garrydias <[hidden email]> wrote:

> Dear friends
>
> I´m facing the folowing problem: I have 2 producers systems sending messages
> to 1 jms queue that is consumed by 2 consumers systems. Ignore the VM where
> they are.
>
> #a) a producerSystem_1.war has the folowing camelContext:
>
> <camelContext id="producerSystem-1-camelContext">
>   ...
>   <route>
>     <from uri="direct:whatever" />
>     <to uri="activemq:common_queue" />
>   </route>
>   ...
> </camelContext >
>
>
> #b) a producerSystem_2.war has the folowing camelContext:
>
> <camelContext id="producerSystem-2-camelContext">
>   ...
>   <route>
>     <from uri="direct:whatever" />
>     <to uri="activemq:common_queue" />
>   </route>
>   ...
> </camelContext >
>
>
> #c) a consumerSystem_1.war has the folowing camelContext:
>
> <camelContext id="context_1">
>   <route>
>     <from uri="activemq:common_queue" />
>     <to uri="bean:consumer_a" />
>   </route>
> </camelContext >
>
>
> #d) a consumerSystem_2.war has the folowing camelContext:
>
> <camelContext id="context_2">
>   <route>
>     <from uri="activemq:common_queue" />
>     <to uri="bean:consumer_b" />
>   </route>
> </camelContext >
>
>
> The VM where these applications are running is not important.
>
> My *desired behavior *is:
> - Messages produced in #a must consumed in c#
> - Messages produced in #b must consumed in d#
>
> The *actual behavior* is:
> - Messages produced in #a sometimes is consumed in c# and sometimes in d#
> - Messages produced in #b sometimes is consumed in c# and sometimes in d#
>
> I understand that JMS endpoints were designed to execute in the *actual
> behavior*. But in this case, I need to run *my desired *behavior.
>
> There´s a way to do this using correlationIds (not send/receive approach)?
>
> Thanx a lot
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Using-correlationid-to-isolate-jms-queues-in-multiple-camelContexts-tp5719202.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

Christian Mueller
Administrator
+1

Sent from a mobile device
Am 12.09.2012 19:44 schrieb "Pontus Ullgren" <[hidden email]>:

> Hello,
>
> The way you see it behave is the way it should behave. If you have
> competing consumers on a queue they will consume messages in that way.
>
> If I interpreted your requirement correctly you have messages from a#
> that can only be consumed by c# and messages from b# that can only be
> consumed by d#.
> In that case you should really use two different queues for this one
> between each pair of producer and consumer.
>
> Adding extra queues in activemq is simple and very low cost when it
> comes to resources. So I would advice you to solve it this way.
>
> If you for some reason still want to use one queue (legacy, crazy
> customer requirement or let's say that you are building a example for
> bad architecture ;-P) then you can set a header on the messages in the
> producer and then use a selector in the consumer. See the
> documentation on the Camel JMS component[1] and the JMS component unit
> test.
>
> But again: Use two queues.
>
> [1] http://camel.apache.org/jms.html
> [2]
> https://svn.apache.org/repos/asf/camel/trunk/components/camel-jms/src/test/java/org/apache/camel/component/jms/JmsSelectorTest.java
> Best regards
> Pontus Ullgren
>
>
> On Wed, Sep 12, 2012 at 7:18 PM, garrydias <[hidden email]> wrote:
> > Dear friends
> >
> > I´m facing the folowing problem: I have 2 producers systems sending
> messages
> > to 1 jms queue that is consumed by 2 consumers systems. Ignore the VM
> where
> > they are.
> >
> > #a) a producerSystem_1.war has the folowing camelContext:
> >
> > <camelContext id="producerSystem-1-camelContext">
> >   ...
> >   <route>
> >     <from uri="direct:whatever" />
> >     <to uri="activemq:common_queue" />
> >   </route>
> >   ...
> > </camelContext >
> >
> >
> > #b) a producerSystem_2.war has the folowing camelContext:
> >
> > <camelContext id="producerSystem-2-camelContext">
> >   ...
> >   <route>
> >     <from uri="direct:whatever" />
> >     <to uri="activemq:common_queue" />
> >   </route>
> >   ...
> > </camelContext >
> >
> >
> > #c) a consumerSystem_1.war has the folowing camelContext:
> >
> > <camelContext id="context_1">
> >   <route>
> >     <from uri="activemq:common_queue" />
> >     <to uri="bean:consumer_a" />
> >   </route>
> > </camelContext >
> >
> >
> > #d) a consumerSystem_2.war has the folowing camelContext:
> >
> > <camelContext id="context_2">
> >   <route>
> >     <from uri="activemq:common_queue" />
> >     <to uri="bean:consumer_b" />
> >   </route>
> > </camelContext >
> >
> >
> > The VM where these applications are running is not important.
> >
> > My *desired behavior *is:
> > - Messages produced in #a must consumed in c#
> > - Messages produced in #b must consumed in d#
> >
> > The *actual behavior* is:
> > - Messages produced in #a sometimes is consumed in c# and sometimes in d#
> > - Messages produced in #b sometimes is consumed in c# and sometimes in d#
> >
> > I understand that JMS endpoints were designed to execute in the *actual
> > behavior*. But in this case, I need to run *my desired *behavior.
> >
> > There´s a way to do this using correlationIds (not send/receive
> approach)?
> >
> > Thanx a lot
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Using-correlationid-to-isolate-jms-queues-in-multiple-camelContexts-tp5719202.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

garrydias
This post has NOT been accepted by the mailing list yet.
Thanx, for your replies :)

Pontus, you interpreted well my requirements.

My problem is really due a legacy system.

In my first solution I used a dynamicRouter in a more decopled enviroment but that solution lost in performance.
The second solution should be that in my last post but it will lose in jms queue amounts and maintainability. If I put a selector maybe its results could be the same as using a dynamicRouter =(  I´ll create POCs in my enviroment to test DynamicRouter X Selector and check the best approach for my specific problems.


Now, consider this similar problem:
- Messages produced in route-a are consumed in route-b and sometimes in route-d.
- Messages produced in route-c are consumed in route-b and sometimes in route-d.

#a) a System_1.war has the folowing camelContext:

<camelContext id="System-1-camelContext">
  ...
  <route id="route-a">
    <from uri="direct:whatever" />
    <to uri="activemq:common_queue" />
  </route>
 
  <route id="route-b">
    <from uri="activemq:common_queue" />
    <to uri="activemq:specific_queue_a" />
  </route> 
</camelContext>


#b) a System_2.war has the folowing camelContext:

<camelContext id="System-2-camelContext">
  ...
  <route id="route-c">
    <from uri="direct:whatever" />
    <to uri="activemq:common_queue" />
  </route>
 
  <route id="route-d">
    <from uri="activemq:common_queue" />
    <to uri="activemq:specific_queue_b" />
  </route>
</camelContext>

Can I isolate the production and consumption in the same camelContext? I know I can do it using SEDA (wich preserves my essencial asynchrony) but I lose message durability. Again, can I preserve my activemq component and isolate production and consumption within camelContexts using correlation id approach or something else?

Thanx





2012/9/12 Christian Mueller [via Camel] <[hidden email]>
+1

Sent from a mobile device
Am 12.09.2012 19:44 schrieb "Pontus Ullgren" <[hidden email]>:

> Hello,
>
> The way you see it behave is the way it should behave. If you have
> competing consumers on a queue they will consume messages in that way.
>
> If I interpreted your requirement correctly you have messages from a#
> that can only be consumed by c# and messages from b# that can only be
> consumed by d#.
> In that case you should really use two different queues for this one
> between each pair of producer and consumer.
>
> Adding extra queues in activemq is simple and very low cost when it
> comes to resources. So I would advice you to solve it this way.
>
> If you for some reason still want to use one queue (legacy, crazy
> customer requirement or let's say that you are building a example for
> bad architecture ;-P) then you can set a header on the messages in the
> producer and then use a selector in the consumer. See the
> documentation on the Camel JMS component[1] and the JMS component unit
> test.
>
> But again: Use two queues.
>
> [1] http://camel.apache.org/jms.html
> [2]
> https://svn.apache.org/repos/asf/camel/trunk/components/camel-jms/src/test/java/org/apache/camel/component/jms/JmsSelectorTest.java
> Best regards

> Pontus Ullgren
>
>
> On Wed, Sep 12, 2012 at 7:18 PM, garrydias <[hidden email]> wrote:
> > Dear friends
> >
> > I´m facing the folowing problem: I have 2 producers systems sending
> messages
> > to 1 jms queue that is consumed by 2 consumers systems. Ignore the VM
> where
> > they are.
> >
> > #a) a producerSystem_1.war has the folowing camelContext:
> >
> > <camelContext id="producerSystem-1-camelContext">
> >   ...
> >   <route>
> >     <from uri="direct:whatever" />
> >     <to uri="activemq:common_queue" />
> >   </route>
> >   ...
> > </camelContext >
> >
> >
> > #b) a producerSystem_2.war has the folowing camelContext:
> >
> > <camelContext id="producerSystem-2-camelContext">
> >   ...
> >   <route>
> >     <from uri="direct:whatever" />
> >     <to uri="activemq:common_queue" />
> >   </route>
> >   ...
> > </camelContext >
> >
> >
> > #c) a consumerSystem_1.war has the folowing camelContext:
> >
> > <camelContext id="context_1">
> >   <route>
> >     <from uri="activemq:common_queue" />
> >     <to uri="bean:consumer_a" />
> >   </route>
> > </camelContext >
> >
> >
> > #d) a consumerSystem_2.war has the folowing camelContext:
> >
> > <camelContext id="context_2">
> >   <route>
> >     <from uri="activemq:common_queue" />
> >     <to uri="bean:consumer_b" />
> >   </route>
> > </camelContext >
> >
> >
> > The VM where these applications are running is not important.
> >
> > My *desired behavior *is:
> > - Messages produced in #a must consumed in c#
> > - Messages produced in #b must consumed in d#
> >
> > The *actual behavior* is:
> > - Messages produced in #a sometimes is consumed in c# and sometimes in d#
> > - Messages produced in #b sometimes is consumed in c# and sometimes in d#
> >
> > I understand that JMS endpoints were designed to execute in the *actual
> > behavior*. But in this case, I need to run *my desired *behavior.
> >
> > There´s a way to do this using correlationIds (not send/receive
> approach)?
> >
> > Thanx a lot
> >
> >
> >
> > --
> > View this message in context:
> http://camel.465427.n5.nabble.com/Using-correlationid-to-isolate-jms-queues-in-multiple-camelContexts-tp5719202.html
> > Sent from the Camel - Users mailing list archive at Nabble.com.
>



To unsubscribe from Using correlationid to isolate jms queues in multiple camelContexts, click here.
NAML



--
"Não espero nada cair do céu. Derrubo logo!!

Garry Dias
Rio de Janeiro, RJ, Brasil
Contatos:
   Celular:   55 21 7135-8755
   E-mails: [hidden email]    
                [hidden email](MSN Messenger)
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

Claus Ibsen-2
In reply to this post by garrydias
Hi

You can also use JMS message groups
http://activemq.apache.org/how-do-message-groups-compare-to-selectors.html
http://activemq.apache.org/message-groups.html

Then the producer can set the group id, and make sure there is only
one active, and the same, consumer for each unique group.



On Wed, Sep 12, 2012 at 7:18 PM, garrydias <[hidden email]> wrote:

> Dear friends
>
> I´m facing the folowing problem: I have 2 producers systems sending messages
> to 1 jms queue that is consumed by 2 consumers systems. Ignore the VM where
> they are.
>
> #a) a producerSystem_1.war has the folowing camelContext:
>
> <camelContext id="producerSystem-1-camelContext">
>   ...
>   <route>
>     <from uri="direct:whatever" />
>     <to uri="activemq:common_queue" />
>   </route>
>   ...
> </camelContext >
>
>
> #b) a producerSystem_2.war has the folowing camelContext:
>
> <camelContext id="producerSystem-2-camelContext">
>   ...
>   <route>
>     <from uri="direct:whatever" />
>     <to uri="activemq:common_queue" />
>   </route>
>   ...
> </camelContext >
>
>
> #c) a consumerSystem_1.war has the folowing camelContext:
>
> <camelContext id="context_1">
>   <route>
>     <from uri="activemq:common_queue" />
>     <to uri="bean:consumer_a" />
>   </route>
> </camelContext >
>
>
> #d) a consumerSystem_2.war has the folowing camelContext:
>
> <camelContext id="context_2">
>   <route>
>     <from uri="activemq:common_queue" />
>     <to uri="bean:consumer_b" />
>   </route>
> </camelContext >
>
>
> The VM where these applications are running is not important.
>
> My *desired behavior *is:
> - Messages produced in #a must consumed in c#
> - Messages produced in #b must consumed in d#
>
> The *actual behavior* is:
> - Messages produced in #a sometimes is consumed in c# and sometimes in d#
> - Messages produced in #b sometimes is consumed in c# and sometimes in d#
>
> I understand that JMS endpoints were designed to execute in the *actual
> behavior*. But in this case, I need to run *my desired *behavior.
>
> There´s a way to do this using correlationIds (not send/receive approach)?
>
> Thanx a lot
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/Using-correlationid-to-isolate-jms-queues-in-multiple-camelContexts-tp5719202.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
FuseSource
Email: [hidden email]
Web: http://fusesource.com
Twitter: davsclaus, fusenews
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

Ashwin Karpe
This post was updated on .
Hi Claus,

Not sure whether message groups will work if the consumers are different applications since message groups are more about ordered processing of messages via a single consumer. It is not meant ot perform content based routing of messages to specific consumer and cannot not distinguish between consumers and could cause the broker to push messages towards the wrong application.

There are 2 options here
        - Adding a Camel CBR to the mix
                  or
        - Adding Virtual Destinations (if the JMS provider is ActiveMQ)

The Virtual Destination could direct messages to the right consumer when a user defined header is used by the message producer.

In all cases, the assumption is that the consumers are looking at their own queues and the producers produce to common/virtual queue or topic.

My quick 2 cents worth.

Cheers,

Ashwin...

 
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
Red Hat
http://www.redhat.com

Blog:http://opensourceknowledge.blogspot.com
---------------------------------------------------------
Reply | Threaded
Open this post in threaded view
|

Re: Using correlationid to isolate jms queues in multiple camelContexts

garrydias
This post has NOT been accepted by the mailing list yet.
Thanx for your replies.

I´ll POC Virtual destination.

CBR is not cool for me because the consumed messages are deleted from activemq queue before passing by content check instructions.

By now, selector is running fine. I had to develop a new layer to set the selectors. My workflow won´t be faster than I expected but will be faster than using a DynamicRouter reading a nextDestination header from exchanges.

Thanx again.




2012/9/13 Ashwin Karpe [via Camel] <[hidden email]>
Hi Claus,

Not sure whether message groups will work if the consumers are different applications since message groups are more about ordered processing of messages via a single consumer. It is not meant ot perform content based routing of messages to specific consumer and cannot not distinguish between consumers and could cause the broker to push messages towards the wrong application.

There are 2 options here
        - Adding a Camel CBR to the mix
                  or
        - Adding Virtual Destinations (if the JMS provider is ActiveMQ)

The Virtual Destination could direct messages to the right consumer.

In all cases, the assumption is that the consumers are looking at their own queues and the producers produce to common/virtual queue or topic.

My quick 2 cents worth.

Cheers,

Ashwin...

 
---------------------------------------------------------
Ashwin Karpe
Apache Camel Committer & Sr Principal Consultant
FUSESource (a Progress Software Corporation subsidiary)
http://fusesource.com

Blog:http://opensourceknowledge.blogspot.com
---------------------------------------------------------



To unsubscribe from Using correlationid to isolate jms queues in multiple camelContexts, click here.
NAML



--
"Não espero nada cair do céu. Derrubo logo!!

Garry Dias
Rio de Janeiro, RJ, Brasil
Contatos:
   Celular:   55 21 7135-8755
   E-mails: [hidden email]    
                [hidden email](MSN Messenger)