REST endpoint to rabbitmq queue connection timeout

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

REST endpoint to rabbitmq queue connection timeout

Valdis

Hi All,

 

Would anyone know how to configure the timeout values for the below connection pool or the connection pool itself? Been having some issues lately in a test environment with this and the timeout check seems to happen every few milliseconds producing huge amounts of log files.

We’re suspecting it’s a network issue that’s causing this as there is nothing much between the REST endpoint on the buffer queue and only if there is some decent load on the endpoint (10req/sec or so):

 

rest(restEndpoint).post(postEndpoint)
     .type(FeedbackRequest.
class)
     .route()
.process(
this::process)
     .startupOrder(Integer.
MAX_VALUE - 2)
.marshal().json(JsonLibrary.
Jackson, String.class)
.to(
feedbackEndpoint)
     .endRest();

 

where process() just sets a persistence header for RabbitMQ to flush the message to disk:

 

exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);

 

 

2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed delivery for (MessageId: ID-VM1-1538665977944-0-101280 on ExchangeId: ID-VM1-1538665977944-0-101281). Exhausted after delivery attempt: 1 caught: java.util.NoSuchElementException: Timeout waiting for idle object

 

Message History

---------------------------------------------------------------------------------------------------------------------------------------

RouteId              ProcessorId          Processor                                                                        Elapsed (ms)

[route24           ] [route24           ] [restlet:///cch/feedback?restletMethods=POST                                   ] [      3798]

[route24           ] [process22         ] [Processor@0x37c71429                                                          ] [         0]

[route24           ] [marshal1          ] [marshal[org.apache.camel.model.dataformat.JsonDataFormat@c940d2]              ] [         0]

[route24           ] [route24           ] [rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec] [      3782]

 

Stacktrace

---------------------------------------------------------------------------------------------------------------------------------------

java.util.NoSuchElementException: Timeout waiting for idle object

                at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)

                at org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQProducer.java:73)

                at org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(RabbitMQProducer.java:285)

                at org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(RabbitMQProducer.java:272)

                at org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQProducer.java:195)

                at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)

                at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)

                at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)

                at org.apache.camel.processor.Pipeline.process(Pipeline.java:138)

                at org.apache.camel.processor.Pipeline.process(Pipeline.java:101)

                at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)

                at org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)

                at org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletConsumer.java:68)

                at org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBasedRouter.java:58)

 

 

Thanks,

Valdis


Vhi Group DAC (Vhi) is a holding company for insurance and healthcare services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi Health Services DAC and Vhi Investments DAC. Vhi Healthcare DAC trading as Vhi Healthcare and Vhi Insurance DAC trading as Vhi Insurance are regulated by the Central Bank of Ireland. Vhi Healthcare is tied to Vhi Insurance DAC for health insurance in Ireland which is underwritten by Vhi Insurance DAC. Vhi Healthcare is tied to Zurich Life Assurance plc for Vhi Life Term Insurance and Vhi Mortgage Protection which is underwritten by Zurich Life Assurance plc. Vhi Healthcare is tied to Collinson Insurance Services Limited for MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental Insurance which are underwritten by Great Lakes Insurance SE, UK branch and for Vhi Canada Cover and Vhi International Health Insurance which are underwritten by Astrenska Insurance Limited. For more information about the Vhi Group please go to: https://www.vhi.ie/about-vhi.


Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh seirbhísí árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi Healthcare DAC, Vhi Insurance DAC, Vhi Health Services DAC agus Vhi Investments DAC. Déanann Banc Ceannais na hÉireann rialáil ar Vhi Healthcare DAC, ag trádáil dó mar Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance. Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas sláinte in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi Healthcare ceangailte le Zurich Life Assurance plc le haghaidh Árachais Saoil de chuid Vhi agus Árachas Cosanta Morgáiste de chuid Vhi atá frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare ceangailte le Collinson Insurance Services Limited le haghaidh Árachas Taistil Ilturais agus Turasóirí Mála Droma agus Árachas Fiaclóireachta de chuid Vhi atá frithgheallta ag Great Lakes Insurance SE, UK branch agus le haghaidh Clúdach Cheanada de chuid Vhi agus Árachas Sláinte Idirnáisiúnta de chuid Vhi atá frithgheallta ag Astrenska Insurance Limited. Chun tuilleadh faisnéise a fháil faoi Ghrúpa Vhi, tabhair cuairt ar: https://www.vhi.ie/about-vhi.

This e-mail and any files transmitted with it contain information which may be confidential and which may also be privileged and is intended solely for the use of the individual or entity to whom it is addressed. Unless you are the intended recipient you may not copy or use it, or disclose it to anyone else. Any opinions expressed are that of the individual and not necessarily that of the Vhi Group. If you have received this e-mail in error please notify the sender by return.







Reply | Threaded
Open this post in threaded view
|

Re: REST endpoint to rabbitmq queue connection timeout

Zoran Regvart-2
Hi Valdis,
I think you're looking for `channelPoolMaxSize` and
`channelPoolMaxWait` parameters[1]. From the snippet, I guess, you
would set them on the `feedbackEndpoint`, and from the message history
I see that you already set the `connectionFactory`, so you would
configure these much in the same way.

zoran

[1] https://github.com/apache/camel/blob/master/components/camel-rabbitmq/src/main/docs/rabbitmq-component.adoc

On Thu, Oct 4, 2018 at 6:15 PM, Valdis Andersons
<[hidden email]> wrote:

> Hi All,
>
>
>
> Would anyone know how to configure the timeout values for the below
> connection pool or the connection pool itself? Been having some issues
> lately in a test environment with this and the timeout check seems to happen
> every few milliseconds producing huge amounts of log files.
>
> We’re suspecting it’s a network issue that’s causing this as there is
> nothing much between the REST endpoint on the buffer queue and only if there
> is some decent load on the endpoint (10req/sec or so):
>
>
>
> rest(restEndpoint).post(postEndpoint)
>      .type(FeedbackRequest.class)
>      .route()
> .process(this::process)
>      .startupOrder(Integer.MAX_VALUE - 2)
> .marshal().json(JsonLibrary.Jackson, String.class)
> .to(feedbackEndpoint)
>      .endRest();
>
>
>
> where process() just sets a persistence header for RabbitMQ to flush the
> message to disk:
>
>
>
> exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);
>
>
>
>
>
> 2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR
> o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed delivery
> for (MessageId: ID-VM1-1538665977944-0-101280 on ExchangeId:
> ID-VM1-1538665977944-0-101281). Exhausted after delivery attempt: 1 caught:
> java.util.NoSuchElementException: Timeout waiting for idle object
>
>
>
> Message History
>
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> RouteId              ProcessorId          Processor
> Elapsed (ms)
>
> [route24           ] [route24           ]
> [restlet:///cch/feedback?restletMethods=POST
> ] [      3798]
>
> [route24           ] [process22         ] [Processor@0x37c71429
> ] [         0]
>
> [route24           ] [marshal1          ]
> [marshal[org.apache.camel.model.dataformat.JsonDataFormat@c940d2]
> ] [         0]
>
> [route24           ] [route24           ]
> [rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec]
> [      3782]
>
>
>
> Stacktrace
>
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> java.util.NoSuchElementException: Timeout waiting for idle object
>
>                 at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQProducer.java:73)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(RabbitMQProducer.java:285)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(RabbitMQProducer.java:272)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQProducer.java:195)
>
>                 at
> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:148)
>
>                 at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:548)
>
>                 at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
>
>                 at
> org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
>
>                 at
> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>
>                 at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:201)
>
>                 at
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:97)
>
>                 at
> org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletConsumer.java:68)
>
>                 at
> org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBasedRouter.java:58)
>
>
>
>
>
> Thanks,
>
> Valdis
>
>
> Vhi Group DAC (Vhi) is a holding company for insurance and healthcare
> services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi Health
> Services DAC and Vhi Investments DAC. Vhi Healthcare DAC trading as Vhi
> Healthcare and Vhi Insurance DAC trading as Vhi Insurance are regulated by
> the Central Bank of Ireland. Vhi Healthcare is tied to Vhi Insurance DAC for
> health insurance in Ireland which is underwritten by Vhi Insurance DAC. Vhi
> Healthcare is tied to Zurich Life Assurance plc for Vhi Life Term Insurance
> and Vhi Mortgage Protection which is underwritten by Zurich Life Assurance
> plc. Vhi Healthcare is tied to Collinson Insurance Services Limited for
> MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental
> Insurance which are underwritten by Great Lakes Insurance SE, UK branch and
> for Vhi Canada Cover and Vhi International Health Insurance which are
> underwritten by Astrenska Insurance Limited. For more information about the
> Vhi Group please go to: https://www.vhi.ie/about-vhi.
>
>
> Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh seirbhísí
> árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi Healthcare DAC,
> Vhi Insurance DAC, Vhi Health Services DAC agus Vhi Investments DAC. Déanann
> Banc Ceannais na hÉireann rialáil ar Vhi Healthcare DAC, ag trádáil dó mar
> Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance.
> Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas
> sláinte in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi
> Healthcare ceangailte le Zurich Life Assurance plc le haghaidh Árachais
> Saoil de chuid Vhi agus Árachas Cosanta Morgáiste de chuid Vhi atá
> frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare ceangailte le
> Collinson Insurance Services Limited le haghaidh Árachas Taistil Ilturais
> agus Turasóirí Mála Droma agus Árachas Fiaclóireachta de chuid Vhi atá
> frithgheallta ag Great Lakes Insurance SE, UK branch agus le haghaidh
> Clúdach Cheanada de chuid Vhi agus Árachas Sláinte Idirnáisiúnta de chuid
> Vhi atá frithgheallta ag Astrenska Insurance Limited. Chun tuilleadh
> faisnéise a fháil faoi Ghrúpa Vhi, tabhair cuairt ar:
> https://www.vhi.ie/about-vhi.
>
> This e-mail and any files transmitted with it contain information which may
> be confidential and which may also be privileged and is intended solely for
> the use of the individual or entity to whom it is addressed. Unless you are
> the intended recipient you may not copy or use it, or disclose it to anyone
> else. Any opinions expressed are that of the individual and not necessarily
> that of the Vhi Group. If you have received this e-mail in error please
> notify the sender by return.
>
>
>
>
>
>
>
>



--
Zoran Regvart
Reply | Threaded
Open this post in threaded view
|

RE: REST endpoint to rabbitmq queue connection timeout

Valdis
Thank you very much Zoran, I'll try it out with those parameters and see how it goes.


Valdis

-----Original Message-----
From: Zoran Regvart [mailto:[hidden email]]
Sent: 05 October 2018 10:45
To: [hidden email]
Subject: Re: REST endpoint to rabbitmq queue connection timeout

Hi Valdis,
I think you're looking for `channelPoolMaxSize` and `channelPoolMaxWait` parameters[1]. From the snippet, I guess, you would set them on the `feedbackEndpoint`, and from the message history I see that you already set the `connectionFactory`, so you would configure these much in the same way.

zoran

[1] https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVSbEt5EF_ffZc_A&s=33&u=https%3a%2f%2fgithub%2ecom%2fapache%2fcamel%2fblob%2fmaster%2fcomponents%2fcamel-rabbitmq%2fsrc%2fmain%2fdocs%2frabbitmq-component%2eadoc

On Thu, Oct 4, 2018 at 6:15 PM, Valdis Andersons <[hidden email]> wrote:

> Hi All,
>
>
>
> Would anyone know how to configure the timeout values for the below
> connection pool or the connection pool itself? Been having some issues
> lately in a test environment with this and the timeout check seems to
> happen every few milliseconds producing huge amounts of log files.
>
> We’re suspecting it’s a network issue that’s causing this as there is
> nothing much between the REST endpoint on the buffer queue and only if
> there is some decent load on the endpoint (10req/sec or so):
>
>
>
> rest(restEndpoint).post(postEndpoint)
>      .type(FeedbackRequest.class)
>      .route()
> .process(this::process)
>      .startupOrder(Integer.MAX_VALUE - 2)
> .marshal().json(JsonLibrary.Jackson, String.class)
> .to(feedbackEndpoint)
>      .endRest();
>
>
>
> where process() just sets a persistence header for RabbitMQ to flush
> the message to disk:
>
>
>
> exchange.getIn().setHeader("rabbitmq.DELIVERY_MODE", 2);
>
>
>
>
>
> 2018-10-04 17:01:43,275 [https-jsse-nio-8443-exec-65] ERROR
> o.a.c.p.DefaultErrorHandler ID-VM1-1538665977944-0-101281 - Failed
> delivery for (MessageId: ID-VM1-1538665977944-0-101280 on ExchangeId:
> ID-VM1-1538665977944-0-101281). Exhausted after delivery attempt: 1 caught:
> java.util.NoSuchElementException: Timeout waiting for idle object
>
>
>
> Message History
>
> ----------------------------------------------------------------------
> -----------------------------------------------------------------
>
> RouteId              ProcessorId          Processor
> Elapsed (ms)
>
> [route24           ] [route24           ]
> [restlet:///cch/feedback?restletMethods=POST
> ] [      3798]
>
> [route24           ] [process22         ] [Processor@0x37c71429
> ] [         0]
>
> [route24           ] [marshal1          ]
> [marshal[org.apache.camel.model.dataformat.JsonDataFormat@c940d2]
> ] [         0]
>
> [route24           ] [route24           ]
> [rabbitmq://vm1:5671/feedbackBufferExchange?connectionFactory=rabbitConnec]
> [      3782]
>
>
>
> Stacktrace
>
> ----------------------------------------------------------------------
> -----------------------------------------------------------------
>
> java.util.NoSuchElementException: Timeout waiting for idle object
>
>                 at
> org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObj
> ectPool.java:1174)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.execute(RabbitMQP
> roducer.java:73)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.basicPublish(Rabb
> itMQProducer.java:285)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.processInOnly(Rab
> bitMQProducer.java:272)
>
>                 at
> org.apache.camel.component.rabbitmq.RabbitMQProducer.process(RabbitMQP
> roducer.java:195)
>
>                 at
> org.apache.camel.processor.SendProcessor.process(SendProcessor.java:14
> 8)
>
>                 at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryEr
> rorHandler.java:548)
>
>                 at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInterna
> lProcessor.java:201)
>
>                 at
> org.apache.camel.processor.Pipeline.process(Pipeline.java:138)
>
>                 at
> org.apache.camel.processor.Pipeline.process(Pipeline.java:101)
>
>                 at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInterna
> lProcessor.java:201)
>
>                 at
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyn
> cProcessor.java:97)
>
>                 at
> org.apache.camel.component.restlet.RestletConsumer$1.handle(RestletCon
> sumer.java:68)
>
>                 at
> org.apache.camel.component.restlet.MethodBasedRouter.handle(MethodBase
> dRouter.java:58)
>
>
>
>
>
> Thanks,
>
> Valdis
>
>
> Vhi Group DAC (Vhi) is a holding company for insurance and healthcare
> services, which include Vhi Healthcare DAC, Vhi Insurance DAC, Vhi
> Health Services DAC and Vhi Investments DAC. Vhi Healthcare DAC
> trading as Vhi Healthcare and Vhi Insurance DAC trading as Vhi
> Insurance are regulated by the Central Bank of Ireland. Vhi Healthcare
> is tied to Vhi Insurance DAC for health insurance in Ireland which is
> underwritten by Vhi Insurance DAC. Vhi Healthcare is tied to Zurich
> Life Assurance plc for Vhi Life Term Insurance and Vhi Mortgage
> Protection which is underwritten by Zurich Life Assurance plc. Vhi
> Healthcare is tied to Collinson Insurance Services Limited for
> MultiTrip Travel Insurance, Backpacker Travel Insurance and Vhi Dental
> Insurance which are underwritten by Great Lakes Insurance SE, UK
> branch and for Vhi Canada Cover and Vhi International Health Insurance
> which are underwritten by Astrenska Insurance Limited. For more
> information about the Vhi Group please go to:
> https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVS
> bEt5BB8L6UPrw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi
>
>
> Tá Vhi Group DAC (Vhi) ina chuideachta sealbhaíochta le haghaidh
> seirbhísí árachais agus seirbhísí cúram sláinte, lena n-áirítear Vhi
> Healthcare DAC, Vhi Insurance DAC, Vhi Health Services DAC agus Vhi
> Investments DAC. Déanann Banc Ceannais na hÉireann rialáil ar Vhi
> Healthcare DAC, ag trádáil dó mar Vhi Healthcare, agus ar Vhi Insurance DAC, ag trádáil dó mar Vhi Insurance.
> Tá Vhi Healthcare ceangailte le Vhi Insurance DAC le haghaidh árachas
> sláinte in Éirinn, rud atá frithgheallta ag Vhi Insurance DAC. Tá Vhi
> Healthcare ceangailte le Zurich Life Assurance plc le haghaidh
> Árachais Saoil de chuid Vhi agus Árachas Cosanta Morgáiste de chuid
> Vhi atá frithgheallta ag Zurich Life Assurance plc. Tá Vhi Healthcare
> ceangailte le Collinson Insurance Services Limited le haghaidh Árachas
> Taistil Ilturais agus Turasóirí Mála Droma agus Árachas Fiaclóireachta
> de chuid Vhi atá frithgheallta ag Great Lakes Insurance SE, UK branch
> agus le haghaidh Clúdach Cheanada de chuid Vhi agus Árachas Sláinte
> Idirnáisiúnta de chuid Vhi atá frithgheallta ag Astrenska Insurance
> Limited. Chun tuilleadh faisnéise a fháil faoi Ghrúpa Vhi, tabhair cuairt ar:
> https://scanmail.trustwave.com/?c=6600&d=prK32zpd-TqvMeNzuqm1P9sl2_MVS
> bEt5BB8L6UPrw&s=33&u=https%3a%2f%2fwww%2evhi%2eie%2fabout-vhi
>
> This e-mail and any files transmitted with it contain information
> which may be confidential and which may also be privileged and is
> intended solely for the use of the individual or entity to whom it is
> addressed. Unless you are the intended recipient you may not copy or
> use it, or disclose it to anyone else. Any opinions expressed are that
> of the individual and not necessarily that of the Vhi Group. If you
> have received this e-mail in error please notify the sender by return.
>
>
>
>
>
>
>
>



--
Zoran Regvart