Looking for a transactional solution

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Looking for a transactional solution

Brenneis, Steve
I hope this is the right place to ask this question. If not, maybe someone can direct me to the right place.

I have an application that reads messages from a queue, passes them off to a content-based router, which then passes them to various validation and transformation routes. At the end of each series of routes the transformed, validated messages are placed onto another queue.

The problem to be solved is that the messages must be delivered once and only once to the outgoing queue. The solution to this would seem to be a queue-to-queue XA transaction, but I can't find anything that indicates that a transaction could cross route boundaries.

Is there such a solution available? Thanks for any help with this.

Steve Brenneis

This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and notify us immediately.
Reply | Threaded
Open this post in threaded view
|

Re: Looking for a transactional solution

gdaswani
Steve,

           The transactional EIP supports crossing multiple routes as long
as each one of those routes are transactionally aware (transaction
demarcation) and your JMS connection factory is configured properly and
set-up to use a transaction manager.   I personally use camel under an OSGI
container (karaf & blueprint)) and transaction demarcation stuff works just
fine with the transaction manager that is feature installable with KARAF
(JTA).    If you are using something like spring boot, plenty of JTA
transaction managers available (atomikos, bitronix).  The JTA transaction
manager will allow you to read from one type of message broker (e.q.
ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one
transaction boundary.

https://camel.apache.org/components/3.4.x/eips/transactional-client.html

For example.

<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
    <property name="transactionManager" ref="jmsTransactionManager"/></bean>
<bean id="PROPAGATION_REQUIRES_NEW"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
    <property name="transactionManager" ref="jmsTransactionManager"/>
    <property name="propagationBehaviorName"
value="PROPAGATION_REQUIRES_NEW"/></bean>


from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");

from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");

from("writeToQ").transacted(PROPAGATION_REQUIRED");





On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
[hidden email]> wrote:

> I hope this is the right place to ask this question. If not, maybe someone
> can direct me to the right place.
>
> I have an application that reads messages from a queue, passes them off to
> a content-based router, which then passes them to various validation and
> transformation routes. At the end of each series of routes the transformed,
> validated messages are placed onto another queue.
>
> The problem to be solved is that the messages must be delivered once and
> only once to the outgoing queue. The solution to this would seem to be a
> queue-to-queue XA transaction, but I can't find anything that indicates
> that a transaction could cross route boundaries.
>
> Is there such a solution available? Thanks for any help with this.
>
> Steve Brenneis
>
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and notify us
> immediately.
>
Reply | Threaded
Open this post in threaded view
|

RE: [External] Re: Looking for a transactional solution

Brenneis, Steve
Thank you for your answer George.

We are using the XML DSL. When we try something like this:

<route id="route 1">
  <from uri="jms:queue:queuein/>
  <transacted ref="PROPAGATION_REQUIRES_NEW"/>
  <process ref="process1/>
  <to uri="direct:nextRoute1/>
</route>

<route id="route 2>
  <from uri="direct:nextRoute1"/>
  <transacted ref="PROPAGATION_REQUIRED"/>
  <process ref="process2"/>
  <to uri="direct:nextRoute2"/>
</route>

<route id="route 3">
  <from uri="direct:nextRoute2"/>
  <transacted ref="PROPAGATION_REQUIRED"/>
  <process ref="process3">
  <to ref="jms:queue:queueout/>
</route>

If process2 throws an exception, we see that the transaction is rolled back with redelivered = false and the message has disappeared. However, if process1 throws an exception, the transaction is rolled back with redelivered = true and the message is back on the incoming queue.

This is what I meant by a transaction spanning routes. It seems like this approach should work, but we must be using it wrong.

-----Original Message-----
From: George Daswani <[hidden email]>
Sent: Wednesday, April 7, 2021 4:20 PM
To: [hidden email]
Subject: [External] Re: Looking for a transactional solution



WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
 Examine the URL and use caution BEFORE clicking links or opening any attachments.
Steve,

           The transactional EIP supports crossing multiple routes as long as each one of those routes are transactionally aware (transaction
demarcation) and your JMS connection factory is configured properly and
set-up to use a transaction manager.   I personally use camel under an OSGI
container (karaf & blueprint)) and transaction demarcation stuff works just fine with the transaction manager that is feature installable with KARAF
(JTA).    If you are using something like spring boot, plenty of JTA
transaction managers available (atomikos, bitronix).  The JTA transaction manager will allow you to read from one type of message broker (e.q.
ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one transaction boundary.

https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html

For example.

<bean id="PROPAGATION_REQUIRED"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
    <property name="transactionManager" ref="jmsTransactionManager"/></bean>
<bean id="PROPAGATION_REQUIRES_NEW"
class="org.apache.camel.spring.spi.SpringTransactionPolicy">
    <property name="transactionManager" ref="jmsTransactionManager"/>
    <property name="propagationBehaviorName"
value="PROPAGATION_REQUIRES_NEW"/></bean>


from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");

from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");

from("writeToQ").transacted(PROPAGATION_REQUIRED");





On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve < [hidden email]> wrote:

> I hope this is the right place to ask this question. If not, maybe
> someone can direct me to the right place.
>
> I have an application that reads messages from a queue, passes them
> off to a content-based router, which then passes them to various
> validation and transformation routes. At the end of each series of
> routes the transformed, validated messages are placed onto another queue.
>
> The problem to be solved is that the messages must be delivered once
> and only once to the outgoing queue. The solution to this would seem
> to be a queue-to-queue XA transaction, but I can't find anything that
> indicates that a transaction could cross route boundaries.
>
> Is there such a solution available? Thanks for any help with this.
>
> Steve Brenneis
>
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and notify us immediately.
>
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and notify us immediately.
Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: Looking for a transactional solution

Zheng Feng
 Hi Brenneis,

Have you ever tried to set "transacted=true" with the incoming queue ? Just
like <from uri="jms:queue:queuein?transacted=true/>
I think this will use a local transaction with JmsTransactionManager and
rollback the message when an exception is thrown.

Hope this is helpful !


On Fri, Apr 9, 2021 at 12:36 AM Brenneis, Steve <
[hidden email]> wrote:

> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
>   <from uri="jms:queue:queuein/>
>   <transacted ref="PROPAGATION_REQUIRES_NEW"/>
>   <process ref="process1/>
>   <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
>   <from uri="direct:nextRoute1"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process2"/>
>   <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
>   <from uri="direct:nextRoute2"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process3">
>   <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared. However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like this
> approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <[hidden email]>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: [hidden email]
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
>  Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
>            The transactional EIP supports crossing multiple routes as long
> as each one of those routes are transactionally aware (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager.   I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable with
> KARAF
> (JTA).    If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix).  The JTA transaction
> manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one
> transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager" ref="jmsTransactionManager"/>
>     <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> [hidden email]> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything that
> > indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete the
> > e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and notify us
> immediately.
>
Reply | Threaded
Open this post in threaded view
|

Re: [External] Re: Looking for a transactional solution

gdaswani
In reply to this post by Brenneis, Steve
Hello Steve,

      It seems to work for me, let me know if you want to see a sample
maven project for it.   Are you sure that your "jms" component is
configured correctly and has a transactionManager set and has transacted =
true?

<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="jmsConnectionFactory" />
<property name="transactionManager" ref="transactionManager" />
<property name="transacted" value="true" />
</bean>

The sample project I created has a scheduled route that writes to the INPUT
queue the current date and time every 10 seconds.  There's a route that
reads from the queue wrapped in a transaction (propagation_requires_new),
has a processor that displays the message on the log, then calls another
route wrapped in a transaction (propagation_required) which writes to an
outgoing queue.  At the end of the second route, it checks to see if the
date/time written ends with "30" and if so, throws an exception (which will
roll back the write to the outgoing queue, bubbling up to the reading
queue).   The TransactionErrorHandler (as long as there's no redeliveries
set on the camel side) automatically rolled back the transaction and the
message broker automatically redelivers it.

Just a caveat though, some message brokers (e.q. I used ActiveMQ with the
"jms" component) defines a default edelivery policy.   ActiveMQ for example
only redeliveries up to six times if you don't override it with
"jms.redeliveryPolicy.maximumRedeliveries" URL connection property.


On Thu, Apr 8, 2021 at 9:27 AM Brenneis, Steve <
[hidden email]> wrote:

> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
>   <from uri="jms:queue:queuein/>
>   <transacted ref="PROPAGATION_REQUIRES_NEW"/>
>   <process ref="process1/>
>   <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
>   <from uri="direct:nextRoute1"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process2"/>
>   <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
>   <from uri="direct:nextRoute2"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process3">
>   <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared. However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like this
> approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <[hidden email]>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: [hidden email]
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
>  Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
>            The transactional EIP supports crossing multiple routes as long
> as each one of those routes are transactionally aware (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager.   I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable with
> KARAF
> (JTA).    If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix).  The JTA transaction
> manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in one
> transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F%2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client.html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager" ref="jmsTransactionManager"/>
>     <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direct:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> [hidden email]> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything that
> > indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete the
> > e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or confidential. If
> you are not the intended recipient, please delete the e-mail and notify us
> immediately.
>
Reply | Threaded
Open this post in threaded view
|

RE: [External] Re: Looking for a transactional solution

Brenneis, Steve
Hi George.

It appears that when we use the embedded ActiveMQ server, i.e. the vm protocol, it will not negotiate a transaction with IBM MQ. The standalone ActiveMQ broker seems to work.

Thank you for your help.

-----Original Message-----
From: George Daswani <[hidden email]>
Sent: Thursday, April 8, 2021 11:39 PM
To: [hidden email]
Subject: Re: [External] Re: Looking for a transactional solution



WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
 Examine the URL and use caution BEFORE clicking links or opening any attachments.
Hello Steve,

      It seems to work for me, let me know if you want to see a sample
maven project for it.   Are you sure that your "jms" component is
configured correctly and has a transactionManager set and has transacted = true?

<bean id="jms" class="org.apache.camel.component.jms.JmsComponent">
<property name="connectionFactory" ref="jmsConnectionFactory" /> <property name="transactionManager" ref="transactionManager" /> <property name="transacted" value="true" /> </bean>

The sample project I created has a scheduled route that writes to the INPUT queue the current date and time every 10 seconds.  There's a route that reads from the queue wrapped in a transaction (propagation_requires_new), has a processor that displays the message on the log, then calls another route wrapped in a transaction (propagation_required) which writes to an outgoing queue.  At the end of the second route, it checks to see if the date/time written ends with "30" and if so, throws an exception (which will roll back the write to the outgoing queue, bubbling up to the reading
queue).   The TransactionErrorHandler (as long as there's no redeliveries
set on the camel side) automatically rolled back the transaction and the message broker automatically redelivers it.

Just a caveat though, some message brokers (e.q. I used ActiveMQ with the
"jms" component) defines a default edelivery policy.   ActiveMQ for example
only redeliveries up to six times if you don't override it with "jms.redeliveryPolicy.maximumRedeliveries" URL connection property.


On Thu, Apr 8, 2021 at 9:27 AM Brenneis, Steve < [hidden email]> wrote:

> Thank you for your answer George.
>
> We are using the XML DSL. When we try something like this:
>
> <route id="route 1">
>   <from uri="jms:queue:queuein/>
>   <transacted ref="PROPAGATION_REQUIRES_NEW"/>
>   <process ref="process1/>
>   <to uri="direct:nextRoute1/>
> </route>
>
> <route id="route 2>
>   <from uri="direct:nextRoute1"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process2"/>
>   <to uri="direct:nextRoute2"/>
> </route>
>
> <route id="route 3">
>   <from uri="direct:nextRoute2"/>
>   <transacted ref="PROPAGATION_REQUIRED"/>
>   <process ref="process3">
>   <to ref="jms:queue:queueout/>
> </route>
>
> If process2 throws an exception, we see that the transaction is rolled
> back with redelivered = false and the message has disappeared.
> However, if
> process1 throws an exception, the transaction is rolled back with
> redelivered = true and the message is back on the incoming queue.
>
> This is what I meant by a transaction spanning routes. It seems like
> this approach should work, but we must be using it wrong.
>
> -----Original Message-----
> From: George Daswani <[hidden email]>
> Sent: Wednesday, April 7, 2021 4:20 PM
> To: [hidden email]
> Subject: [External] Re: Looking for a transactional solution
>
>
>
> WARNING: This message was received from an EXTERNAL EMAIL ADDRESS.
>  Examine the URL and use caution BEFORE clicking links or opening any
> attachments.
> Steve,
>
>            The transactional EIP supports crossing multiple routes as
> long as each one of those routes are transactionally aware
> (transaction
> demarcation) and your JMS connection factory is configured properly and
> set-up to use a transaction manager.   I personally use camel under an OSGI
> container (karaf & blueprint)) and transaction demarcation stuff works
> just fine with the transaction manager that is feature installable
> with KARAF
> (JTA).    If you are using something like spring boot, plenty of JTA
> transaction managers available (atomikos, bitronix).  The JTA
> transaction manager will allow you to read from one type of message broker (e.q.
> ActiveMQ) and write to another type of message broker (e.q. IBM MQ) in
> one transaction boundary.
>
>
> https://clicktime.symantec.com/3NL8rtk3CaQa9bJreF5FDn7Vc?u=https%3A%2F
> %2Fcamel.apache.org%2Fcomponents%2F3.4.x%2Feips%2Ftransactional-client
> .html
>
> For example.
>
> <bean id="PROPAGATION_REQUIRED"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager"
> ref="jmsTransactionManager"/></bean>
> <bean id="PROPAGATION_REQUIRES_NEW"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>     <property name="transactionManager" ref="jmsTransactionManager"/>
>     <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRES_NEW"/></bean>
>
>
>
> from("readFromQueue").transacted("PROPAGATION_REQUIRES_NEW").to("direc
> t:doSomeProcessing").to("writeToQ");
>
> from("doSomeProcessing").transacted("PROPAGATION_REQUIRED");
>
> from("writeToQ").transacted(PROPAGATION_REQUIRED");
>
>
>
>
>
> On Wed, Apr 7, 2021 at 12:28 PM Brenneis, Steve <
> [hidden email]> wrote:
>
> > I hope this is the right place to ask this question. If not, maybe
> > someone can direct me to the right place.
> >
> > I have an application that reads messages from a queue, passes them
> > off to a content-based router, which then passes them to various
> > validation and transformation routes. At the end of each series of
> > routes the transformed, validated messages are placed onto another queue.
> >
> > The problem to be solved is that the messages must be delivered once
> > and only once to the outgoing queue. The solution to this would seem
> > to be a queue-to-queue XA transaction, but I can't find anything
> > that indicates that a transaction could cross route boundaries.
> >
> > Is there such a solution available? Thanks for any help with this.
> >
> > Steve Brenneis
> >
> > This e-mail may contain information that is privileged or
> > confidential. If you are not the intended recipient, please delete
> > the e-mail and notify us immediately.
> >
> This e-mail may contain information that is privileged or
> confidential. If you are not the intended recipient, please delete the
> e-mail and notify us immediately.
>
This e-mail may contain information that is privileged or confidential. If you are not the intended recipient, please delete the e-mail and notify us immediately.