ActiveMQ concurrentConsumers vs multiple connections

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

ActiveMQ concurrentConsumers vs multiple connections

slew77

We are finding that under load that a single activemq endpoint with many concurrent consumers can't keep up with the volume of traffic. However, if we split the concurrent consumers between several identical routes it performs much better.

E.g. if we configure this:

        <route>
            <from uri="activemq:queue:test?concurrentConsumers=100"/>
            <transacted/>
            <to uri="log:test?level=DEBUG"/>
        </route>

The test queue quickly backs up.

However, if we configure 10 identical routes:
        <route>
            <from uri="activemq:queue:test?concurrentConsumers=10"/>
            <transacted/>
            <to uri="log:test?level=DEBUG"/>
        </route>
        <route>
            <from uri="activemq:queue:test?concurrentConsumers=10"/>
            <transacted/>
            <to uri="log:test?level=DEBUG"/>
        </route>
        <route>
            <from uri="activemq:queue:test?concurrentConsumers=10"/>
            <transacted/>
            <to uri="log:test?level=DEBUG"/>
        </route>
etc

Then the same load is handled without backing up.

---

This is the activemq configuration:

   
    <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
        <property name="connectionFactory" ref="poolConnectionFactory"/>
       
        <property name="transacted" value="true"/>
       
        <property name="transactionManager" ref="jmsTransactionManager"/>
        <property name="cacheLevelName" value="CACHE_CONSUMER" />
    </bean>

   
    <bean id="jmsTransactionManager" class="org.springframework.jms.connection.JmsTransactionManager">
        <property name="connectionFactory" ref="poolConnectionFactory"/>
    </bean>

   
    <bean id="poolConnectionFactory" class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start" destroy-method="stop">
        <property name="maxConnections" value="8"/>
        <property name="connectionFactory" ref="jmsConnectionFactory"/>
        <property name="idleTimeout" value="0"/>
    </bean>

    <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
        <property name="brokerURL" value="tcp://localhost:61616"/>
    </bean>

    <bean id="required" class="org.apache.camel.spring.spi.SpringTransactionPolicy">
        <property name="transactionManager" ref="jmsTransactionManager"/>
        <property name="propagationBehaviorName" value="PROPAGATION_REQUIRED"/>
    </bean>


It's much less easy to configure the system when every queue connection must be duplicated. Is there anything that can be tuned for the single route option with more concurrentConsumers?

Thanks,
Steve.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

Claus Ibsen-2
Try setting a lower prefetch size as each consumer has a buffer of
pre-fetched messages. If that buffer is lower then more consumers can
get messages.

The default prefetch is 1000.

On Thu, Feb 18, 2016 at 11:05 AM, slew77 <[hidden email]> wrote:

>
> We are finding that under load that a single activemq endpoint with many
> concurrent consumers can't keep up with the volume of traffic. However, if
> we split the concurrent consumers between several identical routes it
> performs much better.
>
> E.g. if we configure this:
>
>         <route>
>             <from uri="activemq:queue:test?concurrentConsumers=100"/>
>             <transacted/>
>             <to uri="log:test?level=DEBUG"/>
>         </route>
>
> The test queue quickly backs up.
>
> However, if we configure 10 identical routes:
>         <route>
>             <from uri="activemq:queue:test?concurrentConsumers=10"/>
>             <transacted/>
>             <to uri="log:test?level=DEBUG"/>
>         </route>
>         <route>
>             <from uri="activemq:queue:test?concurrentConsumers=10"/>
>             <transacted/>
>             <to uri="log:test?level=DEBUG"/>
>         </route>
>         <route>
>             <from uri="activemq:queue:test?concurrentConsumers=10"/>
>             <transacted/>
>             <to uri="log:test?level=DEBUG"/>
>         </route>
> etc
>
> Then the same load is handled without backing up.
>
> ---
>
> This is the activemq configuration:
>
>
>     <bean id="activemq"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
>         <property name="connectionFactory" ref="poolConnectionFactory"/>
>
>         <property name="transacted" value="true"/>
>
>         <property name="transactionManager" ref="jmsTransactionManager"/>
>         <property name="cacheLevelName" value="CACHE_CONSUMER" />
>     </bean>
>
>
>     <bean id="jmsTransactionManager"
> class="org.springframework.jms.connection.JmsTransactionManager">
>         <property name="connectionFactory" ref="poolConnectionFactory"/>
>     </bean>
>
>
>     <bean id="poolConnectionFactory"
> class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start"
> destroy-method="stop">
>         <property name="maxConnections" value="8"/>
>         <property name="connectionFactory" ref="jmsConnectionFactory"/>
>         <property name="idleTimeout" value="0"/>
>     </bean>
>
>     <bean id="jmsConnectionFactory"
> class="org.apache.activemq.ActiveMQConnectionFactory">
>         <property name="brokerURL" value="tcp://localhost:61616"/>
>     </bean>
>
>     <bean id="required"
> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>         <property name="transactionManager" ref="jmsTransactionManager"/>
>         <property name="propagationBehaviorName"
> value="PROPAGATION_REQUIRED"/>
>     </bean>
>
>
> It's much less easy to configure the system when every queue connection must
> be duplicated. Is there anything that can be tuned for the single route
> option with more concurrentConsumers?
>
> Thanks,
> Steve.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ActiveMQ-concurrentConsumers-vs-multiple-connections-tp5777831.html
> Sent from the Camel - Users mailing list archive at Nabble.com.



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

slew77
This post has NOT been accepted by the mailing list yet.
Hi,

I get the same results.

I wasn't sure how to set this on the client, so set it on the broker:
<policyEntry queue="queue.>" queuePrefetch="0"/>

I tried with values 0, 1, 100 and didn't see a difference.

Thanks,
Steve.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

Quinn Stevenson
In reply to this post by Claus Ibsen-2
One more thing to look at would be the connection pool size.  In the first case, you have 100 concurrent consumers fighting over 8 connections.  In the second case, you only have 30 concurrent consumers fighting over the same 8 connections.

Also, look at the Javadoc for the PooledConnectionFactory ( http://activemq.apache.org/maven/apidocs/org/apache/activemq/jms/pool/PooledConnectionFactory.html <http://activemq.apache.org/maven/apidocs/org/apache/activemq/jms/pool/PooledConnectionFactory.html> ) - I think it may be describing a little of what you’re seeing.

> On Feb 18, 2016, at 3:19 AM, Claus Ibsen <[hidden email]> wrote:
>
> Try setting a lower prefetch size as each consumer has a buffer of
> pre-fetched messages. If that buffer is lower then more consumers can
> get messages.
>
> The default prefetch is 1000.
>
> On Thu, Feb 18, 2016 at 11:05 AM, slew77 <[hidden email]> wrote:
>>
>> We are finding that under load that a single activemq endpoint with many
>> concurrent consumers can't keep up with the volume of traffic. However, if
>> we split the concurrent consumers between several identical routes it
>> performs much better.
>>
>> E.g. if we configure this:
>>
>>        <route>
>>            <from uri="activemq:queue:test?concurrentConsumers=100"/>
>>            <transacted/>
>>            <to uri="log:test?level=DEBUG"/>
>>        </route>
>>
>> The test queue quickly backs up.
>>
>> However, if we configure 10 identical routes:
>>        <route>
>>            <from uri="activemq:queue:test?concurrentConsumers=10"/>
>>            <transacted/>
>>            <to uri="log:test?level=DEBUG"/>
>>        </route>
>>        <route>
>>            <from uri="activemq:queue:test?concurrentConsumers=10"/>
>>            <transacted/>
>>            <to uri="log:test?level=DEBUG"/>
>>        </route>
>>        <route>
>>            <from uri="activemq:queue:test?concurrentConsumers=10"/>
>>            <transacted/>
>>            <to uri="log:test?level=DEBUG"/>
>>        </route>
>> etc
>>
>> Then the same load is handled without backing up.
>>
>> ---
>>
>> This is the activemq configuration:
>>
>>
>>    <bean id="activemq"
>> class="org.apache.activemq.camel.component.ActiveMQComponent">
>>        <property name="connectionFactory" ref="poolConnectionFactory"/>
>>
>>        <property name="transacted" value="true"/>
>>
>>        <property name="transactionManager" ref="jmsTransactionManager"/>
>>        <property name="cacheLevelName" value="CACHE_CONSUMER" />
>>    </bean>
>>
>>
>>    <bean id="jmsTransactionManager"
>> class="org.springframework.jms.connection.JmsTransactionManager">
>>        <property name="connectionFactory" ref="poolConnectionFactory"/>
>>    </bean>
>>
>>
>>    <bean id="poolConnectionFactory"
>> class="org.apache.activemq.pool.PooledConnectionFactory" init-method="start"
>> destroy-method="stop">
>>        <property name="maxConnections" value="8"/>
>>        <property name="connectionFactory" ref="jmsConnectionFactory"/>
>>        <property name="idleTimeout" value="0"/>
>>    </bean>
>>
>>    <bean id="jmsConnectionFactory"
>> class="org.apache.activemq.ActiveMQConnectionFactory">
>>        <property name="brokerURL" value="tcp://localhost:61616"/>
>>    </bean>
>>
>>    <bean id="required"
>> class="org.apache.camel.spring.spi.SpringTransactionPolicy">
>>        <property name="transactionManager" ref="jmsTransactionManager"/>
>>        <property name="propagationBehaviorName"
>> value="PROPAGATION_REQUIRED"/>
>>    </bean>
>>
>>
>> It's much less easy to configure the system when every queue connection must
>> be duplicated. Is there anything that can be tuned for the single route
>> option with more concurrentConsumers?
>>
>> Thanks,
>> Steve.
>>
>>
>>
>> --
>> View this message in context: http://camel.465427.n5.nabble.com/ActiveMQ-concurrentConsumers-vs-multiple-connections-tp5777831.html
>> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

slew77
Hi,

Thanks for the ideas.

From what I can see, the consumer only uses a single connection and is allowed up to 500 (default) sessions and it's these sessions that the concurrent consumers use.

I can see this is the case if I use the activemq web console and inspect the active connections. I can see a session for each concurrent consumer and each session belongs to the same connection. Further each consumer seems to have a fairly even spread of the message load.

I've also now set the prefetch size to 10 using ?jms.prefetchPolicy.queuePrefetch=10 on the broker URI, but doesn't seem to make a difference. I can see the setting has applied from the web console.

Appreciate any more suggestions to check.

Thanks,
Steve.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

Quinn Stevenson
Can you try camel-sjms?  Note that it doesn’t support XA transactions.

> On Feb 18, 2016, at 8:50 AM, slew77 <[hidden email]> wrote:
>
> Hi,
>
> Thanks for the ideas.
>
> From what I can see, the consumer only uses a single connection and is
> allowed up to 500 (default) sessions and it's these sessions that the
> concurrent consumers use.
>
> I can see this is the case if I use the activemq web console and inspect the
> active connections. I can see a session for each concurrent consumer and
> each session belongs to the same connection. Further each consumer seems to
> have a fairly even spread of the message load.
>
> I've also now set the prefetch size to 10 using
> ?jms.prefetchPolicy.queuePrefetch=10 on the broker URI, but doesn't seem to
> make a difference. I can see the setting has applied from the web console.
>
> Appreciate any more suggestions to check.
>
> Thanks,
> Steve.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ActiveMQ-concurrentConsumers-vs-multiple-connections-tp5777831p5777853.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

slew77
Hi,

Sjms works well without local transactions, but as soon as they are turned on it seems even slower than using the activemq component.

In all these cases, one thing I've spotted that seems odd is, if I monitor the In flight count, I only ever see it as 0 or 1, no matter how busy the system is. This happens even with the prefetch count set to 1.

Cheers,
Steve.
Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

Quinn Stevenson
Can  you try SJMS without the Spring transaction config?

I think all you need is an ActiveMQConnectionFactory ( not even a pooled one since SJMS does it’s own internal pooling ) and the “transacted=true” URI parameter.  I think you’ll have to get rid of the <transacted/> as well.

> On Feb 18, 2016, at 9:29 AM, slew77 <[hidden email]> wrote:
>
> Hi,
>
> Sjms works well without local transactions, but as soon as they are turned
> on it seems even slower than using the activemq component.
>
> In all these cases, one thing I've spotted that seems odd is, if I monitor
> the In flight count, I only ever see it as 0 or 1, no matter how busy the
> system is. This happens even with the prefetch count set to 1.
>
> Cheers,
> Steve.
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/ActiveMQ-concurrentConsumers-vs-multiple-connections-tp5777831p5777860.html
> Sent from the Camel - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: ActiveMQ concurrentConsumers vs multiple connections

slew77
In reply to this post by slew77
Needs more testing, but I think this is the culprit:

        <property name="cacheLevelName" value="CACHE_CONSUMER" />

Should be

        <property name="cacheLevelName" value="CACHE_NONE" />

Or take the default value CACHE_AUTO.

It says this here: http://camel.apache.org/jms.html#JMS-transactionCacheLevels