JMS pub/sub (using Camel) performance issue with MQ in XA transactions

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

JMS pub/sub (using Camel) performance issue with MQ in XA transactions

Benoit Fortin
We have a java application doing JMS subscriptions that is using Camel as
its JMS provider.

The application is subscribing to a topic using XA. It consumes 1 message
in the queue, and then closes the XA transaction (each message is part of
an XA transaction). Then the application re-attaches itself to the topic to
start over the same process for each message.

When there are no messages to process, the client waits for 45 seconds
(which is the XA transaction timeout) before closing the request and the XA
transaction and starting a new iteration.

I have analyzed how this process is actually being done using a tcpdump,
and here is what I found:

——

    MQ Client --[XA_START]--> MQ Server

    MQ Client <--[XA_START_REPLY]-- MQ Server


    MQ Client --[SPI (SUBSCRIBE)]--> MQ Server

    MQ Client <--[SPI REPLY]-- MQ Server


    MQ Client --[REQUEST_MSGS]--> MQ Server

    MQ Client --[REQUEST_MSGS]--> MQ Server

    MQ Client <--[NOTIFICATION]-- MQ Server


    MQ Client --[MQCLOSE]--> MQ Server

    MQ Client <--[MQCLOSE_REPLY]-- MQ Server


    MQ Client --[XA_END]--> MQ Server

    MQ Client <--[XA_END_REPLY]-- MQ Server


    MQ Client --[XA_COMMIT]--> MQ Server

    MQ Client <--[XA_COMMIT_REPLY]-- MQ Server

——

This is process is on a durable subscription, so each of the iterations
(even when there is no message to consume) ends up generating I/O on the MQ
server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
MQ keeps trace of its subscribers).

Having a somewhat high number of subscribers doing this process, the MQ
server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
outside of business hours, when there are no messages to consume). This
process also consumes ~3 CPU outside of business hours.

I am a bit puzzled that using XA on pub/sub scales so bad, and I am
wondering if there is any way to implement this solution without doing so
much subscribe operations.
Reply | Threaded
Open this post in threaded view
|

Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions

jbonofre
Hi,

What about using pure ACK mode ?

You can see an example with queue (same can be applied to topic) here:
http://blog.nanthrax.net/?p=820

My point is: do you need XA as AFAIU you are using only JMS as resource
(XA is required when you want to use the same transaction with different
backends like JMS and database for instance) ?

Regards
JB

On 03/09/2019 19:03, Benoit Fortin wrote:

> We have a java application doing JMS subscriptions that is using Camel as
> its JMS provider.
>
> The application is subscribing to a topic using XA. It consumes 1 message
> in the queue, and then closes the XA transaction (each message is part of
> an XA transaction). Then the application re-attaches itself to the topic to
> start over the same process for each message.
>
> When there are no messages to process, the client waits for 45 seconds
> (which is the XA transaction timeout) before closing the request and the XA
> transaction and starting a new iteration.
>
> I have analyzed how this process is actually being done using a tcpdump,
> and here is what I found:
>
> ——
>
>     MQ Client --[XA_START]--> MQ Server
>
>     MQ Client <--[XA_START_REPLY]-- MQ Server
>
>
>     MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
>
>     MQ Client <--[SPI REPLY]-- MQ Server
>
>
>     MQ Client --[REQUEST_MSGS]--> MQ Server
>
>     MQ Client --[REQUEST_MSGS]--> MQ Server
>
>     MQ Client <--[NOTIFICATION]-- MQ Server
>
>
>     MQ Client --[MQCLOSE]--> MQ Server
>
>     MQ Client <--[MQCLOSE_REPLY]-- MQ Server
>
>
>     MQ Client --[XA_END]--> MQ Server
>
>     MQ Client <--[XA_END_REPLY]-- MQ Server
>
>
>     MQ Client --[XA_COMMIT]--> MQ Server
>
>     MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
>
> ——
>
> This is process is on a durable subscription, so each of the iterations
> (even when there is no message to consume) ends up generating I/O on the MQ
> server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
> MQ keeps trace of its subscribers).
>
> Having a somewhat high number of subscribers doing this process, the MQ
> server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> outside of business hours, when there are no messages to consume). This
> process also consumes ~3 CPU outside of business hours.
>
> I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> wondering if there is any way to implement this solution without doing so
> much subscribe operations.
>

--
Jean-Baptiste Onofré
[hidden email]
http://blog.nanthrax.net
Talend - http://www.talend.com
Reply | Threaded
Open this post in threaded view
|

Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions

Benoit Fortin
Hi,

We do need XA as the client application is doing 2 phase commit in (1) the
subscription and (2) a database.

On Tue, Sep 3, 2019 at 14:49 Jean-Baptiste Onofré <[hidden email]> wrote:

> Hi,
>
> What about using pure ACK mode ?
>
> You can see an example with queue (same can be applied to topic) here:
> http://blog.nanthrax.net/?p=820
>
> My point is: do you need XA as AFAIU you are using only JMS as resource
> (XA is required when you want to use the same transaction with different
> backends like JMS and database for instance) ?
>
> Regards
> JB
>
> On 03/09/2019 19:03, Benoit Fortin wrote:
> > We have a java application doing JMS subscriptions that is using Camel as
> > its JMS provider.
> >
> > The application is subscribing to a topic using XA. It consumes 1 message
> > in the queue, and then closes the XA transaction (each message is part of
> > an XA transaction). Then the application re-attaches itself to the topic
> to
> > start over the same process for each message.
> >
> > When there are no messages to process, the client waits for 45 seconds
> > (which is the XA transaction timeout) before closing the request and the
> XA
> > transaction and starting a new iteration.
> >
> > I have analyzed how this process is actually being done using a tcpdump,
> > and here is what I found:
> >
> > ——
> >
> >     MQ Client --[XA_START]--> MQ Server
> >
> >     MQ Client <--[XA_START_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
> >
> >     MQ Client <--[SPI REPLY]-- MQ Server
> >
> >
> >     MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> >     MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> >     MQ Client <--[NOTIFICATION]-- MQ Server
> >
> >
> >     MQ Client --[MQCLOSE]--> MQ Server
> >
> >     MQ Client <--[MQCLOSE_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[XA_END]--> MQ Server
> >
> >     MQ Client <--[XA_END_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[XA_COMMIT]--> MQ Server
> >
> >     MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
> >
> > ——
> >
> > This is process is on a durable subscription, so each of the iterations
> > (even when there is no message to consume) ends up generating I/O on the
> MQ
> > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue,
> where
> > MQ keeps trace of its subscribers).
> >
> > Having a somewhat high number of subscribers doing this process, the MQ
> > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> > outside of business hours, when there are no messages to consume). This
> > process also consumes ~3 CPU outside of business hours.
> >
> > I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> > wondering if there is any way to implement this solution without doing so
> > much subscribe operations.
> >
>
> --
> Jean-Baptiste Onofré
> [hidden email]
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
Reply | Threaded
Open this post in threaded view
|

Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions

Claus Ibsen-2
In reply to this post by Benoit Fortin
camel-jms uses spring jms, so there are many users with this combo
also with IBM MQ and XA.
Look at the many options spring jms has to tune its polling, and also
cache levels you can tweak.
There may be some more idle options you can set to make it "sleep"
longer when there are no messages.
But you can't avoid that spring jms is designed to keep polling for
new messages also when the queue is empty.



On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin <[hidden email]> wrote:

>
> We have a java application doing JMS subscriptions that is using Camel as
> its JMS provider.
>
> The application is subscribing to a topic using XA. It consumes 1 message
> in the queue, and then closes the XA transaction (each message is part of
> an XA transaction). Then the application re-attaches itself to the topic to
> start over the same process for each message.
>
> When there are no messages to process, the client waits for 45 seconds
> (which is the XA transaction timeout) before closing the request and the XA
> transaction and starting a new iteration.
>
> I have analyzed how this process is actually being done using a tcpdump,
> and here is what I found:
>
> ——
>
>     MQ Client --[XA_START]--> MQ Server
>
>     MQ Client <--[XA_START_REPLY]-- MQ Server
>
>
>     MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
>
>     MQ Client <--[SPI REPLY]-- MQ Server
>
>
>     MQ Client --[REQUEST_MSGS]--> MQ Server
>
>     MQ Client --[REQUEST_MSGS]--> MQ Server
>
>     MQ Client <--[NOTIFICATION]-- MQ Server
>
>
>     MQ Client --[MQCLOSE]--> MQ Server
>
>     MQ Client <--[MQCLOSE_REPLY]-- MQ Server
>
>
>     MQ Client --[XA_END]--> MQ Server
>
>     MQ Client <--[XA_END_REPLY]-- MQ Server
>
>
>     MQ Client --[XA_COMMIT]--> MQ Server
>
>     MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
>
> ——
>
> This is process is on a durable subscription, so each of the iterations
> (even when there is no message to consume) ends up generating I/O on the MQ
> server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue, where
> MQ keeps trace of its subscribers).
>
> Having a somewhat high number of subscribers doing this process, the MQ
> server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> outside of business hours, when there are no messages to consume). This
> process also consumes ~3 CPU outside of business hours.
>
> I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> wondering if there is any way to implement this solution without doing so
> much subscribe operations.



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: JMS pub/sub (using Camel) performance issue with MQ in XA transactions

Benoit Fortin
The cache shouldn’t be enabled, as we are using XA (as per Camel’s
recommendation). We have also tuned the polling interval.

That being said, what hurts the I/O subsystem on the MQ server is the
Subscribe operation itself (not the polling for messages).

I am wondering is if there would be any way (inside or outside of Camel) to
have the XA transaction “scoped” on the REQUEST_MSGS requests instead of
the SUBSCRIBE operation? So that each polling doesn’t do a subscribe
operation every time and instead stays attached to an already opened
subscription?

As a reminder, I am attaching below how the XA transaction and subscription
operations are being done (according to a tcpdump I analyzed):

    MQ Client --[XA_START]--> MQ Server

    MQ Client <--[XA_START_REPLY]-- MQ Server


    MQ Client --[SPI (SUBSCRIBE)]--> MQ Server

    MQ Client <--[SPI REPLY]-- MQ Server


    MQ Client --[REQUEST_MSGS]--> MQ Server

    MQ Client --[REQUEST_MSGS]--> MQ Server

    MQ Client <--[NOTIFICATION]-- MQ Server


    MQ Client --[MQCLOSE]--> MQ Server

    MQ Client <--[MQCLOSE_REPLY]-- MQ Server


    MQ Client --[XA_END]--> MQ Server

    MQ Client <--[XA_END_REPLY]-- MQ Server


    MQ Client --[XA_COMMIT]--> MQ Server

    MQ Client <--[XA_COMMIT_REPLY]-- MQ Server



On Wed, Sep 4, 2019 at 05:46 Claus Ibsen <[hidden email]> wrote:

> camel-jms uses spring jms, so there are many users with this combo
> also with IBM MQ and XA.
> Look at the many options spring jms has to tune its polling, and also
> cache levels you can tweak.
> There may be some more idle options you can set to make it "sleep"
> longer when there are no messages.
> But you can't avoid that spring jms is designed to keep polling for
> new messages also when the queue is empty.
>
>
>
> On Tue, Sep 3, 2019 at 7:04 PM Benoit Fortin <[hidden email]>
> wrote:
> >
> > We have a java application doing JMS subscriptions that is using Camel as
> > its JMS provider.
> >
> > The application is subscribing to a topic using XA. It consumes 1 message
> > in the queue, and then closes the XA transaction (each message is part of
> > an XA transaction). Then the application re-attaches itself to the topic
> to
> > start over the same process for each message.
> >
> > When there are no messages to process, the client waits for 45 seconds
> > (which is the XA transaction timeout) before closing the request and the
> XA
> > transaction and starting a new iteration.
> >
> > I have analyzed how this process is actually being done using a tcpdump,
> > and here is what I found:
> >
> > ——
> >
> >     MQ Client --[XA_START]--> MQ Server
> >
> >     MQ Client <--[XA_START_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[SPI (SUBSCRIBE)]--> MQ Server
> >
> >     MQ Client <--[SPI REPLY]-- MQ Server
> >
> >
> >     MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> >     MQ Client --[REQUEST_MSGS]--> MQ Server
> >
> >     MQ Client <--[NOTIFICATION]-- MQ Server
> >
> >
> >     MQ Client --[MQCLOSE]--> MQ Server
> >
> >     MQ Client <--[MQCLOSE_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[XA_END]--> MQ Server
> >
> >     MQ Client <--[XA_END_REPLY]-- MQ Server
> >
> >
> >     MQ Client --[XA_COMMIT]--> MQ Server
> >
> >     MQ Client <--[XA_COMMIT_REPLY]-- MQ Server
> >
> > ——
> >
> > This is process is on a durable subscription, so each of the iterations
> > (even when there is no message to consume) ends up generating I/O on the
> MQ
> > server (I think, mostly on the SYSTEM.DURABLE.SUBSCRIBER.QUEUE queue,
> where
> > MQ keeps trace of its subscribers).
> >
> > Having a somewhat high number of subscribers doing this process, the MQ
> > server ends up I/O bound, with +4000 IOPS on the MQ logs disks (even
> > outside of business hours, when there are no messages to consume). This
> > process also consumes ~3 CPU outside of business hours.
> >
> > I am a bit puzzled that using XA on pub/sub scales so bad, and I am
> > wondering if there is any way to implement this solution without doing so
> > much subscribe operations.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
>