Re: Concurrent consumers from SEDA to CXF

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent consumers from SEDA to CXF

Claus Ibsen-2
Hi

Can you try to explain the numbers a bit more, maybe with some example numbers?

For http connections etc, then CXF has its own set of limitations,
specially if you call the same remote server via the same client, eg
same cxf-endpoint in Camel.
That may have limitations on number of concurrent requests, that can
be lower. So try to look at the CXF documentation about this.

On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <[hidden email]> wrote:

> Greetings,
>
> I have encountered a strange behavior with a simple camel route and I would
> like to ask for help of more experienced users. First, let me quickly show
> the route:
>
> The relevant part:
>             <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>             <to uri="MonitoringService" />
>
> The whole blueprint:
> <?xml version="1.0" encoding="UTF-8"?>
> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi=
> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.apache.
> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0">
>     <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
> schema/blueprint" streamCache="true">
>         <route>
>             <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>             <to uri="MonitoringService" />
>         </route>
>     </camelContext>
>     <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
> MonitoringService/v1" id="MonitoringService" address="http://localhost:8042/
> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName=
> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
>         <cxf:properties>
>             <osgi:entry key="dataFormat" value="PAYLOAD"/>
>         </cxf:properties>
>     </cxf:cxfEndpoint>
> </osgi:blueprint>
>
> Now the expected behavior:
> What I want to achieve is a queue from which 10 parallel threads are
> processing messages, calling the cxf endpoint. The cxf endpoint takes
> several seconds to answer, so I am expecting the speed of processing
> somewhere about 1-2 messages per second.
>
> Observed behavior:
> Using JMX I can see X+N inflight messages in the route, where X is the
> maximum number of concurrent http connections to the cxf endpoint and N is
> the number of concurrent users. For some reason, the workers that are
> waiting for the cxf endpoint to answer are not counted against the
> concurrentConsumers limit. Only the workers that are waiting for the
> connection to be established are counted against the limit of
> concurrentConsumers.
>
> Questions:
> What is the correct behavior for the route?
> If my configuration is incorrect, how can I achieve my target behavior?
>
> The camel version used is 2.20.
>
> If someone would be able to help me, and explain what I did wrong, I would
> very much appreciate it.
>
> thank you for help
> kind regards
> Miroslav Borský



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent consumers from SEDA to CXF

Miroslav Borský
Hi Claus,

I would try to explain. Please ask for the clarification as needed.

In the route in my example, if there will a delayer as the component that
take 2 seconds to proceed, the route will look like this:
<route>
  <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
  <delay><constant>2000</constant></delay>
</route>

there will be 10 inflight messages in this route when the seda queue is
filled. The rate of processing of the messages from the seda queue will be
around 5 messages per second. This is the expected behaviour when I set
concurrentConsumers=10 on the seda queue..

However, when instead of <delay/> there will be the cxf call to a service
that take 2 seconds to answer, things will change. In my runtime
nevironment, I can see 110 inflight messages in the route. Also the rate of
processing of the messages from the seda queue is much higher, around 50 per
second.

The problem I perceive is not the number of concurrent requests for the cxf
endpoint. Problem is the fact, that the combination of seda queue and cxf
endpoint seems to exhaust all the available connections regardless of the
concurrentConsumers setting.

I hope I have provided the example that help you better understand the issue
I am facing. If I have failed in doing so, please bear with me and try to
ask differently.

kind regards
Miroslav Borský


---------- Původní e-mail ----------
Od: Claus Ibsen <[hidden email]>
Komu: [hidden email]
Datum: 30. 7. 2018 17:04:45
Předmět: Re: Concurrent consumers from SEDA to CXF
"Hi

Can you try to explain the numbers a bit more, maybe with some example
numbers?

For http connections etc, then CXF has its own set of limitations,
specially if you call the same remote server via the same client, eg
same cxf-endpoint in Camel.
That may have limitations on number of concurrent requests, that can
be lower. So try to look at the CXF documentation about this.

On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <[hidden email]> wrote:
> Greetings,
>
> I have encountered a strange behavior with a simple camel route and I
would
> like to ask for help of more experienced users. First, let me quickly show

> the route:
>
> The relevant part:
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
>
> The whole blueprint:
> <?xml version="1.0" encoding="UTF-8"?>
> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi
=
> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.
apache.
> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/
v

> 1.0.0">
> <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
> schema/blueprint" streamCache="true">
> <route>
> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
> <to uri="MonitoringService" />
> </route>
> </camelContext>
> <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
> MonitoringService/v1" id="MonitoringService" address="http://localhost:
8042/
> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName
=

> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
> <cxf:properties>
> <osgi:entry key="dataFormat" value="PAYLOAD"/>
> </cxf:properties>
> </cxf:cxfEndpoint>
> </osgi:blueprint>
>
> Now the expected behavior:
> What I want to achieve is a queue from which 10 parallel threads are
> processing messages, calling the cxf endpoint. The cxf endpoint takes
> several seconds to answer, so I am expecting the speed of processing
> somewhere about 1-2 messages per second.
>
> Observed behavior:
> Using JMX I can see X+N inflight messages in the route, where X is the
> maximum number of concurrent http connections to the cxf endpoint and N is

> the number of concurrent users. For some reason, the workers that are
> waiting for the cxf endpoint to answer are not counted against the
> concurrentConsumers limit. Only the workers that are waiting for the
> connection to be established are counted against the limit of
> concurrentConsumers.
>
> Questions:
> What is the correct behavior for the route?
> If my configuration is incorrect, how can I achieve my target behavior?
>
> The camel version used is 2.20.
>
> If someone would be able to help me, and explain what I did wrong, I would

> very much appreciate it.
>
> thank you for help
> kind regards
> Miroslav Borský



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
"
Reply | Threaded
Open this post in threaded view
|

Re: Concurrent consumers from SEDA to CXF

Claus Ibsen-2
Hi

Ah okay its likely because CXF can process asynchronously and you may
not have a limit on the seda/vm queue so it can buffer pending
inflight exchanges.
So try setting cxf to be synchronous, via synchronous=true on its endpoint.

On Tue, Jul 31, 2018 at 2:13 PM, Mirek Borsky <[hidden email]> wrote:

> Hi Claus,
>
> I would try to explain. Please ask for the clarification as needed.
>
> In the route in my example, if there will a delayer as the component that
> take 2 seconds to proceed, the route will look like this:
> <route>
>   <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>   <delay><constant>2000</constant></delay>
> </route>
>
> there will be 10 inflight messages in this route when the seda queue is
> filled. The rate of processing of the messages from the seda queue will be
> around 5 messages per second. This is the expected behaviour when I set
> concurrentConsumers=10 on the seda queue..
>
> However, when instead of <delay/> there will be the cxf call to a service
> that take 2 seconds to answer, things will change. In my runtime
> nevironment, I can see 110 inflight messages in the route. Also the rate of
> processing of the messages from the seda queue is much higher, around 50 per
> second.
>
> The problem I perceive is not the number of concurrent requests for the cxf
> endpoint. Problem is the fact, that the combination of seda queue and cxf
> endpoint seems to exhaust all the available connections regardless of the
> concurrentConsumers setting.
>
> I hope I have provided the example that help you better understand the issue
> I am facing. If I have failed in doing so, please bear with me and try to
> ask differently.
>
> kind regards
> Miroslav Borský
>
>
> ---------- Původní e-mail ----------
> Od: Claus Ibsen <[hidden email]>
> Komu: [hidden email]
> Datum: 30. 7. 2018 17:04:45
> Předmět: Re: Concurrent consumers from SEDA to CXF
> "Hi
>
> Can you try to explain the numbers a bit more, maybe with some example
> numbers?
>
> For http connections etc, then CXF has its own set of limitations,
> specially if you call the same remote server via the same client, eg
> same cxf-endpoint in Camel.
> That may have limitations on number of concurrent requests, that can
> be lower. So try to look at the CXF documentation about this.
>
> On Wed, Jul 25, 2018 at 9:16 PM, Mirek Borsky <[hidden email]> wrote:
>> Greetings,
>>
>> I have encountered a strange behavior with a simple camel route and I
> would
>> like to ask for help of more experienced users. First, let me quickly show
>
>> the route:
>>
>> The relevant part:
>> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>> <to uri="MonitoringService" />
>>
>> The whole blueprint:
>> <?xml version="1.0" encoding="UTF-8"?>
>> <osgi:blueprint xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v
>> 1.0.0 https://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd" xmlns:xsi
> =
>> "http://www.w3.org/2001/XMLSchema-instance" xmlns:cxf="http://camel.
> apache.
>> org/schema/blueprint/cxf" xmlns:osgi="http://www.osgi.org/xmlns/blueprint/
> v
>> 1.0.0">
>> <camelContext id="TestSAMLoggerSeda-ctx" xmlns="http://camel.apache.org/
>> schema/blueprint" streamCache="true">
>> <route>
>> <from uri="vm:sam.logger.to.server?concurrentConsumers=10" />
>> <to uri="MonitoringService" />
>> </route>
>> </camelContext>
>> <cxf:cxfEndpoint xmlns:tns="http://www.talend.org/esb/sam/
>> MonitoringService/v1" id="MonitoringService" address="http://localhost:
> 8042/
>> MonitoringMock" serviceName="tns:MonitoringWebServiceService" endpointName
> =
>> "tns:MonitoringWebServicePort" wsdlURL="deploy/MonitoringService.wsdl">
>> <cxf:properties>
>> <osgi:entry key="dataFormat" value="PAYLOAD"/>
>> </cxf:properties>
>> </cxf:cxfEndpoint>
>> </osgi:blueprint>
>>
>> Now the expected behavior:
>> What I want to achieve is a queue from which 10 parallel threads are
>> processing messages, calling the cxf endpoint. The cxf endpoint takes
>> several seconds to answer, so I am expecting the speed of processing
>> somewhere about 1-2 messages per second.
>>
>> Observed behavior:
>> Using JMX I can see X+N inflight messages in the route, where X is the
>> maximum number of concurrent http connections to the cxf endpoint and N is
>
>> the number of concurrent users. For some reason, the workers that are
>> waiting for the cxf endpoint to answer are not counted against the
>> concurrentConsumers limit. Only the workers that are waiting for the
>> connection to be established are counted against the limit of
>> concurrentConsumers.
>>
>> Questions:
>> What is the correct behavior for the route?
>> If my configuration is incorrect, how can I achieve my target behavior?
>>
>> The camel version used is 2.20.
>>
>> If someone would be able to help me, and explain what I did wrong, I would
>
>> very much appreciate it.
>>
>> thank you for help
>> kind regards
>> Miroslav Borský
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
> "



--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2