ActiveMQ routes performance

[Xenofon Papadopoulos]

Sorry in advance if I should have posted that in ActiveMQ list, but here is
our case.
We are running the same test with two different setups:

Setup 1:
------------
We are using a single ActiveMQ broker and a single Camel with the routes:

from( "jetty://http://..." ).inOnly( "activemq:queue:queue2" )

from("activemq:queue:queue2?maxConcurrentConsumers=100").bean( myProcessor,
"delay" )

The activeMQ component uses
a org.springframework.jms.connection.CachingConnectionFactory

myProcess.delay() delays for 200 ms


Setup 2:
------------
We are using two ActiveMQ brokers and a single Camel with the routes:

from( "jetty://http://..." ).inOnly( "activemq1:queue:queue1" )

from(
"activemq1:queue:queue1?maxConcurrentConsumers=100").inOnly(
"activemq2:queue:queue2"
)

from("activemq2:queue:queue2?maxConcurrentConsumers=100").bean(
myProcessor, "delay" )

Both activeMQ components use their
own org.springframework.jms.connection.CachingConnectionFactory
------------------------

All ActiveMQ brokers are identical, with:

- kahadb
- <policyEntry queue=">" queuePrefetch="20" producerFlowControl="false"
optimizedDispatch="true" useCache="true">
- <memoryUsage limit="4 gb"/>
- <transportConnector name="openwire" uri="tcp://0.0.0.0:xxxx"/>

We test the setup by sending http requests from jmeter with 40 threads and
1200 tps filter. We monitor the enqueue and dequeue rates of all the
queues, and notice that:

Setup 1:
------------
queue2 enqueue: ~900 tps
queue2 dequeue: ~50 tps


Setup 2:
------------
queue1 enqueue: ~1200 tps
queue1 dequeue: ~1200 tps
queue2 enqueue: ~1200 tps
queue2 dequeue: ~150 tps

This means that the 2nd setup causes the brokers to behave much better and
gives us a much higher throughput for our processor.
How can this be explained? Is there any configuration that will let us
achieve the same performance with a single broker?

Thank you
Xenofon

[boday]

tough to say given the unknowns about your setup, but here are a couple of
things to consider...

-for kahadb, set enableJournalDiskSyncs to false to get much better
throughput
-try fewer consumer threads...100 is a lot and you generally have
diminished/negative results with larger thread counts
-tune your CachingConnectionFactory (notably the sessionCacheSize should be
increased from the default)...that said, beware of using this with polling
consumers connections (see
http://stackoverflow.com/questions/5916638/autoreconnect-problem-with-activemq-and-cachingconnectionfactory
)
-alternatively, try using the AMQ PooledConnectionFactory and see if you
get better performance
-see this thread on AMQ performance:
http://stackoverflow.com/questions/7463415/activemq-performance-gotchas-and-precautions


-Ben



On Fri, Aug 3, 2012 at 8:44 AM, Xenofon Papadopoulos <
[hidden email]> wrote:

> Sorry in advance if I should have posted that in ActiveMQ list, but here is
> our case.
> We are running the same test with two different setups:
>
> Setup 1:
> ------------
> We are using a single ActiveMQ broker and a single Camel with the routes:
>
> from( "jetty://http://..." ).inOnly( "activemq:queue:queue2" )
>
> from("activemq:queue:queue2?maxConcurrentConsumers=100").bean( myProcessor,
> "delay" )
>
> The activeMQ component uses
> a org.springframework.jms.connection.CachingConnectionFactory
>
> myProcess.delay() delays for 200 ms
>
>
> Setup 2:
> ------------
> We are using two ActiveMQ brokers and a single Camel with the routes:
>
> from( "jetty://http://..." ).inOnly( "activemq1:queue:queue1" )
>
> from(
> "activemq1:queue:queue1?maxConcurrentConsumers=100").inOnly(
> "activemq2:queue:queue2"
> )
>
> from("activemq2:queue:queue2?maxConcurrentConsumers=100").bean(
> myProcessor, "delay" )
>
> Both activeMQ components use their
> own org.springframework.jms.connection.CachingConnectionFactory
> ------------------------
>
> All ActiveMQ brokers are identical, with:
>
> - kahadb
> - <policyEntry queue=">" queuePrefetch="20" producerFlowControl="false"
> optimizedDispatch="true" useCache="true">
> - <memoryUsage limit="4 gb"/>
> - <transportConnector name="openwire" uri="tcp://0.0.0.0:xxxx"/>
>
> We test the setup by sending http requests from jmeter with 40 threads and
> 1200 tps filter. We monitor the enqueue and dequeue rates of all the
> queues, and notice that:
>
> Setup 1:
> ------------
> queue2 enqueue: ~900 tps
> queue2 dequeue: ~50 tps
>
>
> Setup 2:
> ------------
> queue1 enqueue: ~1200 tps
> queue1 dequeue: ~1200 tps
> queue2 enqueue: ~1200 tps
> queue2 dequeue: ~150 tps
>
> This means that the 2nd setup causes the brokers to behave much better and
> gives us a much higher throughput for our processor.
> How can this be explained? Is there any configuration that will let us
> achieve the same performance with a single broker?
>
> Thank you
> Xenofon
>

[Xenofon Papadopoulos]

Thanks for the tips. I'm trying, however, to understand why there is better
behavior with the intermediate queue, given that all other parameters
(threads etc) remain the same.

Also this is a test; in production we normally use about 800 concurrent
consumers, and the resource use in the servers is pretty low.


On Sat, Aug 4, 2012 at 6:44 AM, Ben O'Day <[hidden email]>wrote:

> tough to say given the unknowns about your setup, but here are a couple of
> things to consider...
>
> -for kahadb, set enableJournalDiskSyncs to false to get much better
> throughput
> -try fewer consumer threads...100 is a lot and you generally have
> diminished/negative results with larger thread counts
> -tune your CachingConnectionFactory (notably the sessionCacheSize should be
> increased from the default)...that said, beware of using this with polling
> consumers connections (see
>
> http://stackoverflow.com/questions/5916638/autoreconnect-problem-with-activemq-and-cachingconnectionfactory
> )
> -alternatively, try using the AMQ PooledConnectionFactory and see if you
> get better performance
> -see this thread on AMQ performance:
>
> http://stackoverflow.com/questions/7463415/activemq-performance-gotchas-and-precautions
>
>
> -Ben
>
>
>
> On Fri, Aug 3, 2012 at 8:44 AM, Xenofon Papadopoulos <
> [hidden email]> wrote:
>
> > Sorry in advance if I should have posted that in ActiveMQ list, but here
> is
> > our case.
> > We are running the same test with two different setups:
> >
> > Setup 1:
> > ------------
> > We are using a single ActiveMQ broker and a single Camel with the routes:
> >
> > from( "jetty://http://..." ).inOnly( "activemq:queue:queue2" )
> >
> > from("activemq:queue:queue2?maxConcurrentConsumers=100").bean(
> myProcessor,
> > "delay" )
> >
> > The activeMQ component uses
> > a org.springframework.jms.connection.CachingConnectionFactory
> >
> > myProcess.delay() delays for 200 ms
> >
> >
> > Setup 2:
> > ------------
> > We are using two ActiveMQ brokers and a single Camel with the routes:
> >
> > from( "jetty://http://..." ).inOnly( "activemq1:queue:queue1" )
> >
> > from(
> > "activemq1:queue:queue1?maxConcurrentConsumers=100").inOnly(
> > "activemq2:queue:queue2"
> > )
> >
> > from("activemq2:queue:queue2?maxConcurrentConsumers=100").bean(
> > myProcessor, "delay" )
> >
> > Both activeMQ components use their
> > own org.springframework.jms.connection.CachingConnectionFactory
> > ------------------------
> >
> > All ActiveMQ brokers are identical, with:
> >
> > - kahadb
> > - <policyEntry queue=">" queuePrefetch="20" producerFlowControl="false"
> > optimizedDispatch="true" useCache="true">
> > - <memoryUsage limit="4 gb"/>
> > - <transportConnector name="openwire" uri="tcp://0.0.0.0:xxxx"/>
> >
> > We test the setup by sending http requests from jmeter with 40 threads
> and
> > 1200 tps filter. We monitor the enqueue and dequeue rates of all the
> > queues, and notice that:
> >
> > Setup 1:
> > ------------
> > queue2 enqueue: ~900 tps
> > queue2 dequeue: ~50 tps
> >
> >
> > Setup 2:
> > ------------
> > queue1 enqueue: ~1200 tps
> > queue1 dequeue: ~1200 tps
> > queue2 enqueue: ~1200 tps
> > queue2 dequeue: ~150 tps
> >
> > This means that the 2nd setup causes the brokers to behave much better
> and
> > gives us a much higher throughput for our processor.
> > How can this be explained? Is there any configuration that will let us
> > achieve the same performance with a single broker?
> >
> > Thank you
> > Xenofon
> >
>

--
*Xenofon Papadopoulos
*
*Upstream
*4 Kastorias & Messinias Street
153 44 Gerakas Attikis
Mob [phone]
DL   [phone]
Fax [phone]
www.upstreamsystems.com

[Sam (Stephen Samuel)]

With the intermediate queue you have a bigger buffer, a 2nd queue.
There might be a bottleneck on certain calls from your initial
endpoint which having the 2nd queue alleviates.

On Sat, Aug 4, 2012 at 9:48 AM, Xenofon Papadopoulos
<[hidden email]> wrote:

> Thanks for the tips. I'm trying, however, to understand why there is better
> behavior with the intermediate queue, given that all other parameters
> (threads etc) remain the same.
>
> Also this is a test; in production we normally use about 800 concurrent
> consumers, and the resource use in the servers is pretty low.
>
>
> On Sat, Aug 4, 2012 at 6:44 AM, Ben O'Day <[hidden email]>wrote:
>
>> tough to say given the unknowns about your setup, but here are a couple of
>> things to consider...
>>
>> -for kahadb, set enableJournalDiskSyncs to false to get much better
>> throughput
>> -try fewer consumer threads...100 is a lot and you generally have
>> diminished/negative results with larger thread counts
>> -tune your CachingConnectionFactory (notably the sessionCacheSize should be
>> increased from the default)...that said, beware of using this with polling
>> consumers connections (see
>>
>> http://stackoverflow.com/questions/5916638/autoreconnect-problem-with-activemq-and-cachingconnectionfactory
>> )
>> -alternatively, try using the AMQ PooledConnectionFactory and see if you
>> get better performance
>> -see this thread on AMQ performance:
>>
>> http://stackoverflow.com/questions/7463415/activemq-performance-gotchas-and-precautions
>>
>>
>> -Ben
>>
>>
>>
>> On Fri, Aug 3, 2012 at 8:44 AM, Xenofon Papadopoulos <
>> [hidden email]> wrote:
>>
>> > Sorry in advance if I should have posted that in ActiveMQ list, but here
>> is
>> > our case.
>> > We are running the same test with two different setups:
>> >
>> > Setup 1:
>> > ------------
>> > We are using a single ActiveMQ broker and a single Camel with the routes:
>> >
>> > from( "jetty://http://..." ).inOnly( "activemq:queue:queue2" )
>> >
>> > from("activemq:queue:queue2?maxConcurrentConsumers=100").bean(
>> myProcessor,
>> > "delay" )
>> >
>> > The activeMQ component uses
>> > a org.springframework.jms.connection.CachingConnectionFactory
>> >
>> > myProcess.delay() delays for 200 ms
>> >
>> >
>> > Setup 2:
>> > ------------
>> > We are using two ActiveMQ brokers and a single Camel with the routes:
>> >
>> > from( "jetty://http://..." ).inOnly( "activemq1:queue:queue1" )
>> >
>> > from(
>> > "activemq1:queue:queue1?maxConcurrentConsumers=100").inOnly(
>> > "activemq2:queue:queue2"
>> > )
>> >
>> > from("activemq2:queue:queue2?maxConcurrentConsumers=100").bean(
>> > myProcessor, "delay" )
>> >
>> > Both activeMQ components use their
>> > own org.springframework.jms.connection.CachingConnectionFactory
>> > ------------------------
>> >
>> > All ActiveMQ brokers are identical, with:
>> >
>> > - kahadb
>> > - <policyEntry queue=">" queuePrefetch="20" producerFlowControl="false"
>> > optimizedDispatch="true" useCache="true">
>> > - <memoryUsage limit="4 gb"/>
>> > - <transportConnector name="openwire" uri="tcp://0.0.0.0:xxxx"/>
>> >
>> > We test the setup by sending http requests from jmeter with 40 threads
>> and
>> > 1200 tps filter. We monitor the enqueue and dequeue rates of all the
>> > queues, and notice that:
>> >
>> > Setup 1:
>> > ------------
>> > queue2 enqueue: ~900 tps
>> > queue2 dequeue: ~50 tps
>> >
>> >
>> > Setup 2:
>> > ------------
>> > queue1 enqueue: ~1200 tps
>> > queue1 dequeue: ~1200 tps
>> > queue2 enqueue: ~1200 tps
>> > queue2 dequeue: ~150 tps
>> >
>> > This means that the 2nd setup causes the brokers to behave much better
>> and
>> > gives us a much higher throughput for our processor.
>> > How can this be explained? Is there any configuration that will let us
>> > achieve the same performance with a single broker?
>> >
>> > Thank you
>> > Xenofon
>> >
>>
>
>
>
> --
> *Xenofon Papadopoulos
> *
> *Upstream
> *4 Kastorias & Messinias Street
> 153 44 Gerakas Attikis
> Mob [phone]
> DL   [phone]
> Fax [phone]
> www.upstreamsystems.com

--
-Sam

[Xenofon Papadopoulos]

Well apparently something is alleviated, but I'm trying to understand
what... The 2 queues are sequential, and the 1st queue is almost always
empty, so I cannot see why I have a bigger buffer.


On Sat, Aug 4, 2012 at 1:17 PM, Sam (Stephen Samuel)
<[hidden email]>wrote:

> With the intermediate queue you have a bigger buffer, a 2nd queue.
> There might be a bottleneck on certain calls from your initial
> endpoint which having the 2nd queue alleviates.
>
> On Sat, Aug 4, 2012 at 9:48 AM, Xenofon Papadopoulos
> <[hidden email]> wrote:
> > Thanks for the tips. I'm trying, however, to understand why there is
> better
> > behavior with the intermediate queue, given that all other parameters
> > (threads etc) remain the same.
> >
> > Also this is a test; in production we normally use about 800 concurrent
> > consumers, and the resource use in the servers is pretty low.
> >
> >
> > On Sat, Aug 4, 2012 at 6:44 AM, Ben O'Day <[hidden email]
> >wrote:
> >
> >> tough to say given the unknowns about your setup, but here are a couple
> of
> >> things to consider...
> >>
> >> -for kahadb, set enableJournalDiskSyncs to false to get much better
> >> throughput
> >> -try fewer consumer threads...100 is a lot and you generally have
> >> diminished/negative results with larger thread counts
> >> -tune your CachingConnectionFactory (notably the sessionCacheSize
> should be
> >> increased from the default)...that said, beware of using this with
> polling
> >> consumers connections (see
> >>
> >>
> http://stackoverflow.com/questions/5916638/autoreconnect-problem-with-activemq-and-cachingconnectionfactory
> >> )
> >> -alternatively, try using the AMQ PooledConnectionFactory and see if you
> >> get better performance
> >> -see this thread on AMQ performance:
> >>
> >>
> http://stackoverflow.com/questions/7463415/activemq-performance-gotchas-and-precautions
> >>
> >>
> >> -Ben
> >>
> >>
> >>
> >> On Fri, Aug 3, 2012 at 8:44 AM, Xenofon Papadopoulos <
> >> [hidden email]> wrote:
> >>
> >> > Sorry in advance if I should have posted that in ActiveMQ list, but
> here
> >> is
> >> > our case.
> >> > We are running the same test with two different setups:
> >> >
> >> > Setup 1:
> >> > ------------
> >> > We are using a single ActiveMQ broker and a single Camel with the
> routes:
> >> >
> >> > from( "jetty://http://..." ).inOnly( "activemq:queue:queue2" )
> >> >
> >> > from("activemq:queue:queue2?maxConcurrentConsumers=100").bean(
> >> myProcessor,
> >> > "delay" )
> >> >
> >> > The activeMQ component uses
> >> > a org.springframework.jms.connection.CachingConnectionFactory
> >> >
> >> > myProcess.delay() delays for 200 ms
> >> >
> >> >
> >> > Setup 2:
> >> > ------------
> >> > We are using two ActiveMQ brokers and a single Camel with the routes:
> >> >
> >> > from( "jetty://http://..." ).inOnly( "activemq1:queue:queue1" )
> >> >
> >> > from(
> >> > "activemq1:queue:queue1?maxConcurrentConsumers=100").inOnly(
> >> > "activemq2:queue:queue2"
> >> > )
> >> >
> >> > from("activemq2:queue:queue2?maxConcurrentConsumers=100").bean(
> >> > myProcessor, "delay" )
> >> >
> >> > Both activeMQ components use their
> >> > own org.springframework.jms.connection.CachingConnectionFactory
> >> > ------------------------
> >> >
> >> > All ActiveMQ brokers are identical, with:
> >> >
> >> > - kahadb
> >> > - <policyEntry queue=">" queuePrefetch="20"
> producerFlowControl="false"
> >> > optimizedDispatch="true" useCache="true">
> >> > - <memoryUsage limit="4 gb"/>
> >> > - <transportConnector name="openwire" uri="tcp://0.0.0.0:xxxx"/>
> >> >
> >> > We test the setup by sending http requests from jmeter with 40 threads
> >> and
> >> > 1200 tps filter. We monitor the enqueue and dequeue rates of all the
> >> > queues, and notice that:
> >> >
> >> > Setup 1:
> >> > ------------
> >> > queue2 enqueue: ~900 tps
> >> > queue2 dequeue: ~50 tps
> >> >
> >> >
> >> > Setup 2:
> >> > ------------
> >> > queue1 enqueue: ~1200 tps
> >> > queue1 dequeue: ~1200 tps
> >> > queue2 enqueue: ~1200 tps
> >> > queue2 dequeue: ~150 tps
> >> >
> >> > This means that the 2nd setup causes the brokers to behave much better
> >> and
> >> > gives us a much higher throughput for our processor.
> >> > How can this be explained? Is there any configuration that will let us
> >> > achieve the same performance with a single broker?
> >> >
> >> > Thank you
> >> > Xenofon
> >> >
> >>
> >
> >
> >
> > --
> > *Xenofon Papadopoulos
> > *
> > *Upstream
> > *4 Kastorias & Messinias Street
> > 153 44 Gerakas Attikis
> > Mob [phone]
> > DL   [phone]
> > Fax [phone]
> > www.upstreamsystems.com
>
>
>
> --
> -Sam
>

--
*Xenofon Papadopoulos
*
*Upstream
*4 Kastorias & Messinias Street
153 44 Gerakas Attikis
Mob [phone]
DL   [phone]
Fax [phone]
www.upstreamsystems.com