JMSxception after 1300 concurrent remote invocations

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

JMSxception after 1300 concurrent remote invocations

asaksena1
Hi,

I'm executing some concurrency tests on Apache Camel 1.5 as a part of its evaluation for our project.
I'm running into JMSException on the Camel Server side if I try to run > 1300 threads. Can someone give me a hint to get around it.

I'm using the JMS Spring Remoting example hosted at http://activemq.apache.org/camel/tutorial-jmsremoting.html with the slight modification in the Camel JMS Producer.

++++++
...    
<bean id="jms" class="org.apache.activemq.camel.component.ActiveMQComponent">
        <property name="brokerURL" value="tcp://localhost:61616"/>
</bean> 
...
++++++


Here are the exceptions


Server:
     [java] [                   VMTransport] Service                        ERROR Async error occurred: javax.jms.JMSException: Could not correlate acknowledgment with dispatched message: MessageAck {
commandId = 6356, responseRequired = false, ackType = 0, consumerId = ID:lap-CKF6GH1-1920-1228931248429-2:0:1310:1, firstMessageId = ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId = ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination = queue://numbers, transactionId = null, messageCount = 24}
     [java] javax.jms.JMSException: Could not correlate acknowledgment with dispatched message: MessageAck {commandId = 6356, responseRequired = false, ackType = 0, consumerId = ID:lap-CKF6GH1-1920-12
28931248429-2:0:1310:1, firstMessageId = ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId = ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination = queue://numbers, transactionId = null, messageCount = 24}
     [java]     at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:304)
     [java]     at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
     [java]     at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
     [java]     at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
     [java]     at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
     [java]     at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
     [java]     at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
     [java]     at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
     [java]     at org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
     [java]     at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
     [java]     at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
     [java]     at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
     [java]     at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
     [java]     at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
     [java]     at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
     [java]     at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
     [java]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
     [java]     at java.lang.Thread.run(Thread.java:619)

++++++++++++++++++++++

Client:
Exception in thread "Thread-930" java.lang.reflect.UndeclaredThrowableException
 at $Proxy23.multiply(Unknown Source)

 at org.apache.camel.example.client.PerfTest.run(PerfTest.java:118)
Caused by: java.lang.reflect.InvocationTargetException
 at org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:57)
 ... 2 more
Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was not received within: 20000 millis on the exchange: Exchange[Message: BeanInvocation public abstract int org.apache.camel.example.server.Multiplier.multiply(int) with [926]]]
 at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:221)
 at org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:54)
 ... 2 more
Reply | Threaded
Open this post in threaded view
|

Re: JMSxception after 1300 concurrent remote invocations

Claus Ibsen-2
Hi

Looks like you get a timeout exception
> Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was
> not received within: 20000 millis on the exchange: Exchange[Message:

There are some options on the jms component you can define to allow
longer timeout values, so the client waits longer.
http://activemq.apache.org/camel/jms.html

requestTimeout defaults to 20 sec. Try with a higher value.


On Wed, Dec 10, 2008 at 7:25 PM, asaksena1 <[hidden email]> wrote:

>
> Hi,
>
> I'm executing some concurrency tests on Apache Camel 1.5 as a part of its
> evaluation for our project.
> I'm running into JMSException on the Camel Server side if I try to run >
> 1300 threads. Can someone give me a hint to get around it.
>
> I'm using the  http://activemq.apache.org/camel/tutorial-jmsremoting.html
> JMS Spring Remoting example  hosted at
> http://activemq.apache.org/camel/tutorial-jmsremoting.html
> http://activemq.apache.org/camel/tutorial-jmsremoting.html  with the slight
> modification in the Camel JMS Producer.
>
> ++++++
> ...
> <bean id="jms"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
>        <property name="brokerURL" value="tcp://localhost:61616"/>
> </bean>
> ...
> ++++++
>
>
> Here are the exceptions
>
>
> Server:
>     [java] [                   VMTransport] Service
> ERROR Async error occurred: javax.jms.JMSException: Could not correlate
> acknowledgment with dispatched message: MessageAck {
> commandId = 6356, responseRequired = false, ackType = 0, consumerId =
> ID:lap-CKF6GH1-1920-1228931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java] javax.jms.JMSException: Could not correlate acknowledgment with
> dispatched message: MessageAck {commandId = 6356, responseRequired = false,
> ackType = 0, consumerId = ID:lap-CKF6GH1-1920-12
> 28931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java]     at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:304)
>     [java]     at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
>     [java]     at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
>     [java]     at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
>     [java]     at
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
>     [java]     at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>     [java]     at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
>     [java]     at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>     [java]     at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     [java]     at java.lang.Thread.run(Thread.java:619)
>
> ++++++++++++++++++++++
>
> Client:
> Exception in thread "Thread-930"
> java.lang.reflect.UndeclaredThrowableException
>  at $Proxy23.multiply(Unknown Source)
>  at org.apache.camel.example.client.PerfTest.run(PerfTest.java:118)
> Caused by: java.lang.reflect.InvocationTargetException
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:57)
>  ... 2 more
> Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was
> not received within: 20000 millis on the exchange: Exchange[Message:
> BeanInvocation public abstract int
> org.apache.camel.example.server.Multiplier.multiply(int) with [926]]]
>  at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:221)
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:54)
>  ... 2 more
> --
> View this message in context: http://www.nabble.com/JMSxception-after-1300-concurrent-remote-invocations-tp20941053s22882p20941053.html
> Sent from the Camel - Development mailing list archive at Nabble.com.
>
>



--

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/
Reply | Threaded
Open this post in threaded view
|

Re: JMSxception after 1300 concurrent remote invocations

jstrachan
In reply to this post by asaksena1
It looks like you're getting a timeout; looks like responses are not
received with 20s. 1300 is quite a large number. Either try increasing
the timeout or reducing the number of threads?

2008/12/10 asaksena1 <[hidden email]>:

>
> Hi,
>
> I'm executing some concurrency tests on Apache Camel 1.5 as a part of its
> evaluation for our project.
> I'm running into JMSException on the Camel Server side if I try to run >
> 1300 threads. Can someone give me a hint to get around it.
>
> I'm using the  http://activemq.apache.org/camel/tutorial-jmsremoting.html
> JMS Spring Remoting example  hosted at
> http://activemq.apache.org/camel/tutorial-jmsremoting.html
> http://activemq.apache.org/camel/tutorial-jmsremoting.html  with the slight
> modification in the Camel JMS Producer.
>
> ++++++
> ...
> <bean id="jms"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
>        <property name="brokerURL" value="tcp://localhost:61616"/>
> </bean>
> ...
> ++++++
>
>
> Here are the exceptions
>
>
> Server:
>     [java] [                   VMTransport] Service
> ERROR Async error occurred: javax.jms.JMSException: Could not correlate
> acknowledgment with dispatched message: MessageAck {
> commandId = 6356, responseRequired = false, ackType = 0, consumerId =
> ID:lap-CKF6GH1-1920-1228931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java] javax.jms.JMSException: Could not correlate acknowledgment with
> dispatched message: MessageAck {commandId = 6356, responseRequired = false,
> ackType = 0, consumerId = ID:lap-CKF6GH1-1920-12
> 28931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java]     at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:304)
>     [java]     at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
>     [java]     at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
>     [java]     at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
>     [java]     at
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
>     [java]     at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>     [java]     at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
>     [java]     at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>     [java]     at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     [java]     at java.lang.Thread.run(Thread.java:619)
>
> ++++++++++++++++++++++
>
> Client:
> Exception in thread "Thread-930"
> java.lang.reflect.UndeclaredThrowableException
>  at $Proxy23.multiply(Unknown Source)
>  at org.apache.camel.example.client.PerfTest.run(PerfTest.java:118)
> Caused by: java.lang.reflect.InvocationTargetException
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:57)
>  ... 2 more
> Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was
> not received within: 20000 millis on the exchange: Exchange[Message:
> BeanInvocation public abstract int
> org.apache.camel.example.server.Multiplier.multiply(int) with [926]]]
>  at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:221)
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:54)
>  ... 2 more
> --
> View this message in context: http://www.nabble.com/JMSxception-after-1300-concurrent-remote-invocations-tp20941053s22882p20941053.html
> Sent from the Camel - Development mailing list archive at Nabble.com.
>
>



--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://fusesource.com/
Reply | Threaded
Open this post in threaded view
|

Re: JMSxception after 1300 concurrent remote invocations

asaksena1
In reply to this post by Claus Ibsen-2
Thanks, I was able to increase the timeout and get responses for more threads.
For the benefit of the group. Here is a brief of my results.

* Invoked 900 threads in ≈ 360 msec
* 3 successive remote invocations / Thread
* Response (quickest) ≈ 1.4 sec [Compare it to 110 msec for normal]
* Response (slowest) ≈ 19 sec

Thanks

Claus Ibsen-2 wrote
Hi

Looks like you get a timeout exception
> Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was
> not received within: 20000 millis on the exchange: Exchange[Message:

There are some options on the jms component you can define to allow
longer timeout values, so the client waits longer.
http://activemq.apache.org/camel/jms.html

requestTimeout defaults to 20 sec. Try with a higher value.


On Wed, Dec 10, 2008 at 7:25 PM, asaksena1 <anurag.saksena@gmail.com> wrote:
>
> Hi,
>
> I'm executing some concurrency tests on Apache Camel 1.5 as a part of its
> evaluation for our project.
> I'm running into JMSException on the Camel Server side if I try to run >
> 1300 threads. Can someone give me a hint to get around it.
>
> I'm using the  http://activemq.apache.org/camel/tutorial-jmsremoting.html
> JMS Spring Remoting example  hosted at
> http://activemq.apache.org/camel/tutorial-jmsremoting.html
> http://activemq.apache.org/camel/tutorial-jmsremoting.html  with the slight
> modification in the Camel JMS Producer.
>
> ++++++
> ...
> <bean id="jms"
> class="org.apache.activemq.camel.component.ActiveMQComponent">
>        <property name="brokerURL" value="tcp://localhost:61616"/>
> </bean>
> ...
> ++++++
>
>
> Here are the exceptions
>
>
> Server:
>     [java] [                   VMTransport] Service
> ERROR Async error occurred: javax.jms.JMSException: Could not correlate
> acknowledgment with dispatched message: MessageAck {
> commandId = 6356, responseRequired = false, ackType = 0, consumerId =
> ID:lap-CKF6GH1-1920-1228931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java] javax.jms.JMSException: Could not correlate acknowledgment with
> dispatched message: MessageAck {commandId = 6356, responseRequired = false,
> ackType = 0, consumerId = ID:lap-CKF6GH1-1920-12
> 28931248429-2:0:1310:1, firstMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:172:1:2, lastMessageId =
> ID:lap-CKF6GH1-2128-1228931452193-0:0:791:1:1, destination =
> queue://numbers, transactionId = null, messageCount = 24}
>     [java]     at
> org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:304)
>     [java]     at
> org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:373)
>     [java]     at
> org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:462)
>     [java]     at
> org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:194)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:74)
>     [java]     at
> org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:85)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:456)
>     [java]     at
> org.apache.activemq.command.MessageAck.visit(MessageAck.java:205)
>     [java]     at
> org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:305)
>     [java]     at
> org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179)
>     [java]     at
> org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:104)
>     [java]     at
> org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
>     [java]     at
> org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:205)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
>     [java]     at
> org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     [java]     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     [java]     at java.lang.Thread.run(Thread.java:619)
>
> ++++++++++++++++++++++
>
> Client:
> Exception in thread "Thread-930"
> java.lang.reflect.UndeclaredThrowableException
>  at $Proxy23.multiply(Unknown Source)
>  at org.apache.camel.example.client.PerfTest.run(PerfTest.java:118)
> Caused by: java.lang.reflect.InvocationTargetException
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:57)
>  ... 2 more
> Caused by: org.apache.camel.ExchangeTimedOutException: The OUT message was
> not received within: 20000 millis on the exchange: Exchange[Message:
> BeanInvocation public abstract int
> org.apache.camel.example.server.Multiplier.multiply(int) with [926]]]
>  at org.apache.camel.component.jms.JmsProducer.process(JmsProducer.java:221)
>  at
> org.apache.camel.component.bean.CamelInvocationHandler.invoke(CamelInvocationHandler.java:54)
>  ... 2 more
> --
> View this message in context: http://www.nabble.com/JMSxception-after-1300-concurrent-remote-invocations-tp20941053s22882p20941053.html
> Sent from the Camel - Development mailing list archive at Nabble.com.
>
>



--

/Claus Ibsen
Apache Camel Committer
Blog: http://davsclaus.blogspot.com/