Getting lots of TIME_WAIT sockets

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Getting lots of TIME_WAIT sockets

DominicTulley
We have camel deployed inside AMQ, providing a routing service for us.
It consumes from an incoming queue, annotates the messages and then sends them on to a second queue from which various engines collect the messages and process them.  Something like this:

Producer -> activemq:IN_QUEUE -> Camel: annotation code -> activemq:OUT_QUEUE -> engines

As we push messages through this we are seeing very large numbers of TIME_WAIT sockets (into the thousands which then leads to a shortage of file descriptors and bad things happening).  

If I completely remove Camel processing from the mix, it all behaves nicely.  If I replace our annotation code with a simple route from IN_QUEUE to OUT_QUEUE we still see all the TIME_WAIT sockets.

It looks awfully like camel is using a new socket for each message it processes.

I understand that TIME_WAIT is a normal part of tcpip but to have so many of them is hopeless.

Around this issue a couple of things occur to me:

1)  Is Camel really opening that many sockets deliberately (using the activemq component) or is it a bug.
2)  What I'd really like to do is not use sockets for this processing.  In my simple view of the world, the two queues involved both have a "socket facing end" and an "internal end".  My camel processing only accesses the internal ends of the queues so I thought I would be able to use a vm/seda type component. Sadly it doesn't seem to work - the messages simply don't get picked up from the IN queue if I specify seda:IN_QUEUE.  Are my expectations wrong here or have I just not got the configuration quite right?

We're currently using a slightly old snapshot build of AMQ 5.1 and a snapshot build of Camel 1.2 (although I've just tried dropping in the 1.3.0 RC 2 build of Camel and that didn't help).  We're running on Java 5.

Thanks in advance for any suggestions or guidance.

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 18/03/2008, DominicTulley <[hidden email]> wrote:

>
>  We have camel deployed inside AMQ, providing a routing service for us.
>  It consumes from an incoming queue, annotates the messages and then sends
>  them on to a second queue from which various engines collect the messages
>  and process them.  Something like this:
>
>  Producer -> activemq:IN_QUEUE -> Camel: annotation code ->
>  activemq:OUT_QUEUE -> engines
>
>  As we push messages through this we are seeing very large numbers of
>  TIME_WAIT sockets (into the thousands which then leads to a shortage of file
>  descriptors and bad things happening).
>
>  If I completely remove Camel processing from the mix, it all behaves nicely.
>  If I replace our annotation code with a simple route from IN_QUEUE to
>  OUT_QUEUE we still see all the TIME_WAIT sockets.
>
>  It looks awfully like camel is using a new socket for each message it
>  processes.
>
>  I understand that TIME_WAIT is a normal part of tcpip but to have so many of
>  them is hopeless.
>
>  Around this issue a couple of things occur to me:
>
>  1)  Is Camel really opening that many sockets deliberately (using the
>  activemq component) or is it a bug.
>  2)  What I'd really like to do is not use sockets for this processing.  In
>  my simple view of the world, the two queues involved both have a "socket
>  facing end" and an "internal end".  My camel processing only accesses the
>  internal ends of the queues so I thought I would be able to use a vm/seda
>  type component. Sadly it doesn't seem to work - the messages simply don't
>  get picked up from the IN queue if I specify seda:IN_QUEUE.  Are my
>  expectations wrong here or have I just not got the configuration quite
>  right?
>
>  We're currently using a slightly old snapshot build of AMQ 5.1 and a
>  snapshot build of Camel 1.2 (although I've just tried dropping in the 1.3.0
>  RC 2 build of Camel and that didn't help).  We're running on Java 5.

Its a rather long discussion - I've added a big read box to the wiki
to try describe it...

http://cwiki.apache.org/CAMEL/jms.html

but basically the quick fix is to specify the cacheLevelName property
to be CACHE_CONSUMER on the ActiveMQComponent or on the endpoint you
consume from.

basically from Spring 2.5.1 or later and Camel 1.3.0 the out of the
box defaults should just work. However due to an earlier spring bug,
Camel took the decision to do a really bad performance implementation
(using a consumer for one message only!) to ensure we maintained
transactional integrity.

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
In reply to this post by DominicTulley
Thanks for the quick response James.

I've tried what I understood by your suggestion (and in fact started scattering the cacheLevelName in other places as well!) but I still see the problem:

from("activemq:IN_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER")
         .setHeader(RoutingStatics.HEADER_HASHANDLER, hasHandlerExpression)
         .setHeader(RoutingStatics.HEADER_ISDOCBASED, isDocBasedExpression)
.to("activemq:OUT_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER");

Is this what you intended, or do you mean on the external consumer (the engines reading from the OUT_QUEUE)?

I've also tried with CACHE_CONNECTION because it sounded like the name might not have changed yet in the version of spring I have in AMQ (2.0.6).

thanks,

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 18/03/2008, DominicTulley <[hidden email]> wrote:
>
>  Thanks for the quick response James.
>
>  I've tried what I understood by your suggestion (and in fact started
>  scattering the cacheLevelName in other places as well!)

You can set this property on the ActiveMQComponent in the spring.xml
too BTW to avoid messing with your URIs.

> but I still see the
>  problem:
>
>  from("activemq:IN_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER")
>          .setHeader(RoutingStatics.HEADER_HASHANDLER, hasHandlerExpression)
>          .setHeader(RoutingStatics.HEADER_ISDOCBASED, isDocBasedExpression)
>  .to("activemq:OUT_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER");
>
>  Is this what you intended,

Yes

> or do you mean on the external consumer (the
>  engines reading from the OUT_QUEUE)?

Is there some other JMS code consuming too?
--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley

I have a total of three java processes running in my "test case":

One is a message producer connecting to the broker at
 tcp://localhost:61616?wireFormat.tcpNoDelayEnabled=true
and refering to the queue
IN_QUEUE?cacheLevelName=CACHE_CONSUMER

One is AMQ with camel embedded, and the previously shown routebuilder

One is a message consumer using the same broker url as the producer
and referring to the queue
OUT_QUEUE?cacheLevelName=CACHE_CONSUMER

I've annotated all the queue URIs with cacheLevelName=CACHE_CONSUMER to no effect.
Both the producer and consumer processes are using activemq connection factories to connect to the broker.
The producer and consumer classes are running as plain java Main methods - no spring or anything like that involved.
An experiment with the producer and consumer on separate machines showed that all the TIME_WAIT sockets are internal connections within AMQ (ie from localhost and to localhost) - not to either the producer or consumer processes.

-Dominic


Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
Just to clear this up - the TIME_WAIT connections are in the ActiveMQ
broker right? And do they keep increasing? You're not using the web
console or anything?

What does your activemq.xml look like?


On 18/03/2008, DominicTulley <[hidden email]> wrote:

>
>
>  I have a total of three java processes running in my "test case":
>
>  One is a message producer connecting to the broker at
>   tcp://localhost:61616?wireFormat.tcpNoDelayEnabled=true
>  and refering to the queue
>  IN_QUEUE?cacheLevelName=CACHE_CONSUMER
>
>  One is AMQ with camel embedded, and the previously shown routebuilder
>
>  One is a message consumer using the same broker url as the producer
>  and referring to the queue
>  OUT_QUEUE?cacheLevelName=CACHE_CONSUMER
>
>  I've annotated all the queue URIs with cacheLevelName=CACHE_CONSUMER to no
>  effect.
>  Both the producer and consumer processes are using activemq connection
>  factories to connect to the broker.
>  The producer and consumer classes are running as plain java Main methods -
>  no spring or anything like that involved.
>  An experiment with the producer and consumer on separate machines showed
>  that all the TIME_WAIT sockets are internal connections within AMQ (ie from
>  localhost and to localhost) - not to either the producer or consumer
>  processes.
>
>  -Dominic
>
>
>
>
>  --
>  View this message in context: http://www.nabble.com/Getting-lots-of-TIME_WAIT-sockets-tp16119896s22882p16125352.html
>
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
If I run with the AMQ broker on one machine and the producer&consumer on a second machine then all the TIME_WAIT sockets are on the AMQ machine (from localhost, to localhost).  So, I'm pretty convinced the cause is there.  Also, if I remove all camel logic and just produce and consume on the same queue, there are no TIME_WAIT sockets (or very few).

The number of sockets is certainly increasing - I'm pushing through around 100 msg/s and seeing around 100 TIME_WAIT sockets appearing per second.  Interestingly the number continues to rise for "a while" after I kill the producer - my assumption is that it is because camel hasn't got through them all yet.  The number of TIME_WAIT sockets stops rising a few seconds after I kill off the producer.

Here's my activemq.xml.
My camel config is all right at the end.
Other than that, the only bit I've deliberately changed is to trim down the network connectors section.

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 18/03/2008, DominicTulley <[hidden email]> wrote:

>
>  If I run with the AMQ broker on one machine and the producer&consumer on a
>  second machine then all the TIME_WAIT sockets are on the AMQ machine (from
>  localhost, to localhost).  So, I'm pretty convinced the cause is there.
>  Also, if I remove all camel logic and just produce and consume on the same
>  queue, there are no TIME_WAIT sockets (or very few).
>
>  The number of sockets is certainly increasing - I'm pushing through around
>  100 msg/s and seeing around 100 TIME_WAIT sockets appearing per second.
>  Interestingly the number continues to rise for "a while" after I kill the
>  producer - my assumption is that it is because camel hasn't got through them
>  all yet.  The number of TIME_WAIT sockets stops rising a few seconds after I
>  kill off the producer.
>
>  Here's my  http://www.nabble.com/file/p16131500/activemq.xml activemq.xml .
>  My camel config is all right at the end.
>  Other than that, the only bit I've deliberately changed is to trim down the
>  network connectors section.

What does the Java code look like for the org.dominic.PrimaryRouteBuilder?


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
In reply to this post by DominicTulley
I grabbed the lastest AMQ 5.1 from svn this morning and built that.
It includes spring 2.5.1 so I thought that would be another way forward.  However, it hasn't helped.
I'm currently trying to get a current camel build (the latest revision in svn doesn't build for some reason so I'm working backwards trying to find a current revision that does).

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
I've managed to build the latest camel from svn as well now and run the two together.

Sending messages at around 100-200/s I see the number of time_Wait sockets reach around 2000 pretty quickly (roughly in the time it takes to send that many messages).  At this point amq starts to report some errors (socket closed) and shortly after, when time_wait socket count reaches around 3500, things really break, with "address already in use" errors from amq.

If we throttle throughput to be less than 2000 messages per two minutes then the TIME_WAIT sockets will disappear promptly enough for the system not to crumble - but we don't really want to have to put throttling in place and 2000 in two minutes doesn't sound like a lot of messages.

Would it be useful to provide a test case at this point?  I'm pretty sure this problem will be reproducible just by taking the sample producer and consumer code from the AMQ distro and changing the config to have a Camel route in the middle...

Thanks,

-Dominic

DominicTulley wrote
I grabbed the lastest AMQ 5.1 from svn this morning and built that.
It includes spring 2.5.1 so I thought that would be another way forward.  However, it hasn't helped.
I'm currently trying to get a current camel build (the latest revision in svn doesn't build for some reason so I'm working backwards trying to find a current revision that does).

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
OK - one last update for now..

This problem is reproducible using the sample ProducerTool and ConsumerTool that come with AMQ.
Just set the producer to produce 10000 messages and set the consumer to listen on a different queue (say TOOL.DEFAULT.ROUTED).

Then add the simple camel route:

  <route>
    <from uri="activemq:TOOL.DEFAULT?cacheLevelName=CACHE_CONSUMER"/>
    <to uri="activemq:TOOL.DEFAULT.ROUTED?cacheLevelName=CACHE_CONSUMER"/>
  </route>

Some of the stack traces (not all) talk about the JmsTemplates in spring (but the stack traces are to do with the deadletterchannel.  Attached is a log file from a failing run of AMQ in case there's something subtle in there...activemq.log

I'd appreciate any further advice or recommendations.

From my original post - I was also wondering if I can use an "in memory" connection to the queue instead of the activemq component.  Is it possible to mix and match the components used to access a queue in this way, so that my camel routes (which are in the same vm as the machine) could use vm/seda/direct style components and my remote clients can use activemq components as they currently do?

Thanks,

-Dominic


DominicTulley wrote
I've managed to build the latest camel from svn as well now and run the two together.

Sending messages at around 100-200/s I see the number of time_Wait sockets reach around 2000 pretty quickly (roughly in the time it takes to send that many messages).  At this point amq starts to report some errors (socket closed) and shortly after, when time_wait socket count reaches around 3500, things really break, with "address already in use" errors from amq.

If we throttle throughput to be less than 2000 messages per two minutes then the TIME_WAIT sockets will disappear promptly enough for the system not to crumble - but we don't really want to have to put throttling in place and 2000 in two minutes doesn't sound like a lot of messages.

Would it be useful to provide a test case at this point?  I'm pretty sure this problem will be reproducible just by taking the sample producer and consumer code from the AMQ distro and changing the config to have a Camel route in the middle...

Thanks,

-Dominic

DominicTulley wrote
I grabbed the lastest AMQ 5.1 from svn this morning and built that.
It includes spring 2.5.1 so I thought that would be another way forward.  However, it hasn't helped.
I'm currently trying to get a current camel build (the latest revision in svn doesn't build for some reason so I'm working backwards trying to find a current revision that does).

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
In reply to this post by jstrachan
I missed this message yesterday.

The primary route builder has this code in it:

from("activemq:IN_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER")
         .setHeader(RoutingStatics.HEADER_HASHANDLER, hasHandlerExpression)
         .setHeader(RoutingStatics.HEADER_ISDOCBASED, isDocBasedExpression)
.to("activemq:OUT_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER");

However, if I get rid of all my camel configuration and instead just use a simple route like that below, it still has the problem.

  <route>
    <from uri="activemq:TOOL.DEFAULT"/>
    <to uri="activemq:TOOL.DEFAULT.ROUTED"/>
  </route>

-Dominic
James.Strachan wrote
On 18/03/2008, DominicTulley <dominic.tulley@telelogic.com> wrote:
>
>  If I run with the AMQ broker on one machine and the producer&consumer on a
>  second machine then all the TIME_WAIT sockets are on the AMQ machine (from
>  localhost, to localhost).  So, I'm pretty convinced the cause is there.
>  Also, if I remove all camel logic and just produce and consume on the same
>  queue, there are no TIME_WAIT sockets (or very few).
>
>  The number of sockets is certainly increasing - I'm pushing through around
>  100 msg/s and seeing around 100 TIME_WAIT sockets appearing per second.
>  Interestingly the number continues to rise for "a while" after I kill the
>  producer - my assumption is that it is because camel hasn't got through them
>  all yet.  The number of TIME_WAIT sockets stops rising a few seconds after I
>  kill off the producer.
>
>  Here's my  http://www.nabble.com/file/p16131500/activemq.xml activemq.xml .
>  My camel config is all right at the end.
>  Other than that, the only bit I've deliberately changed is to trim down the
>  network connectors section.

What does the Java code look like for the org.dominic.PrimaryRouteBuilder?


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
I'm getting a bit confused by all these permuations :)

Whats the easiest way to reproduce the problem you're seeing. Just
take the default activemq.xml and sending a ton of messages to one of
the camel routes in the activemq.xml?


On 20/03/2008, DominicTulley <[hidden email]> wrote:

>
>  I missed this message yesterday.
>
>  The primary route builder has this code in it:
>
>
>  from("activemq:IN_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER")
>          .setHeader(RoutingStatics.HEADER_HASHANDLER, hasHandlerExpression)
>          .setHeader(RoutingStatics.HEADER_ISDOCBASED, isDocBasedExpression)
>  .to("activemq:OUT_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER");
>
>
> However, if I get rid of all my camel configuration and instead just use a
>  simple route like that below, it still has the problem.
>
>   <route>
>     <from uri="activemq:TOOL.DEFAULT"/>
>     <to uri="activemq:TOOL.DEFAULT.ROUTED"/>
>   </route>
>
>  -Dominic
>
>
>  James.Strachan wrote:
>  >
>  > On 18/03/2008, DominicTulley <[hidden email]> wrote:
>  >>
>  >>  If I run with the AMQ broker on one machine and the producer&consumer on
>  >> a
>  >>  second machine then all the TIME_WAIT sockets are on the AMQ machine
>  >> (from
>  >>  localhost, to localhost).  So, I'm pretty convinced the cause is there.
>  >>  Also, if I remove all camel logic and just produce and consume on the
>  >> same
>  >>  queue, there are no TIME_WAIT sockets (or very few).
>  >>
>  >>  The number of sockets is certainly increasing - I'm pushing through
>  >> around
>  >>  100 msg/s and seeing around 100 TIME_WAIT sockets appearing per second.
>  >>  Interestingly the number continues to rise for "a while" after I kill
>  >> the
>  >>  producer - my assumption is that it is because camel hasn't got through
>  >> them
>  >>  all yet.  The number of TIME_WAIT sockets stops rising a few seconds
>  >> after I
>  >>  kill off the producer.
>  >>
>  >>  Here's my  http://www.nabble.com/file/p16131500/activemq.xml
>  >> activemq.xml .
>  >>  My camel config is all right at the end.
>  >>  Other than that, the only bit I've deliberately changed is to trim down
>  >> the
>  >>  network connectors section.
>  >
>  > What does the Java code look like for the org.dominic.PrimaryRouteBuilder?
>  >
>  >
>  > --
>  > James
>  > -------
>  > http://macstrac.blogspot.com/
>  >
>  > Open Source Integration
>  > http://open.iona.com
>  >
>  >
>
>
> --
>  View this message in context: http://www.nabble.com/Getting-lots-of-TIME_WAIT-sockets-tp16119896s22882p16176219.html
>
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
Sorry for the noise - I just end up following my nose trying to find a way round the problem.

The easiest way to reproduce the problem is simply to pump a bunch of messages through a standard camel route defined in activemq.xml.  I see the problem with the standard distro of amq and camel (As built from svn) by just adding a single route (with activemq: as the component) and exercising it.

-Dominic

James.Strachan wrote
I'm getting a bit confused by all these permuations :)

Whats the easiest way to reproduce the problem you're seeing. Just
take the default activemq.xml and sending a ton of messages to one of
the camel routes in the activemq.xml?


On 20/03/2008, DominicTulley <dominic.tulley@telelogic.com> wrote:
>
>  I missed this message yesterday.
>
>  The primary route builder has this code in it:
>
>
>  from("activemq:IN_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER")
>          .setHeader(RoutingStatics.HEADER_HASHANDLER, hasHandlerExpression)
>          .setHeader(RoutingStatics.HEADER_ISDOCBASED, isDocBasedExpression)
>  .to("activemq:OUT_QUEUE?disableReplyTo=true&cacheLevelName=CACHE_CONSUMER");
>
>
> However, if I get rid of all my camel configuration and instead just use a
>  simple route like that below, it still has the problem.
>
>   <route>
>     <from uri="activemq:TOOL.DEFAULT"/>
>     <to uri="activemq:TOOL.DEFAULT.ROUTED"/>
>   </route>
>
>  -Dominic
>
>
>  James.Strachan wrote:
>  >
>  > On 18/03/2008, DominicTulley <dominic.tulley@telelogic.com> wrote:
>  >>
>  >>  If I run with the AMQ broker on one machine and the producer&consumer on
>  >> a
>  >>  second machine then all the TIME_WAIT sockets are on the AMQ machine
>  >> (from
>  >>  localhost, to localhost).  So, I'm pretty convinced the cause is there.
>  >>  Also, if I remove all camel logic and just produce and consume on the
>  >> same
>  >>  queue, there are no TIME_WAIT sockets (or very few).
>  >>
>  >>  The number of sockets is certainly increasing - I'm pushing through
>  >> around
>  >>  100 msg/s and seeing around 100 TIME_WAIT sockets appearing per second.
>  >>  Interestingly the number continues to rise for "a while" after I kill
>  >> the
>  >>  producer - my assumption is that it is because camel hasn't got through
>  >> them
>  >>  all yet.  The number of TIME_WAIT sockets stops rising a few seconds
>  >> after I
>  >>  kill off the producer.
>  >>
>  >>  Here's my  http://www.nabble.com/file/p16131500/activemq.xml
>  >> activemq.xml .
>  >>  My camel config is all right at the end.
>  >>  Other than that, the only bit I've deliberately changed is to trim down
>  >> the
>  >>  network connectors section.
>  >
>  > What does the Java code look like for the org.dominic.PrimaryRouteBuilder?
>  >
>  >
>  > --
>  > James
>  > -------
>  > http://macstrac.blogspot.com/
>  >
>  > Open Source Integration
>  > http://open.iona.com
>  >
>  >
>
>
> --
>  View this message in context: http://www.nabble.com/Getting-lots-of-TIME_WAIT-sockets-tp16119896s22882p16176219.html
>
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>


--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 20/03/2008, DominicTulley <[hidden email]> wrote:
>
>  Sorry for the noise - I just end up following my nose trying to find a way
>  round the problem.

No problem :)

>  The easiest way to reproduce the problem is simply to pump a bunch of
>  messages through a standard camel route defined in activemq.xml.  I see the
>  problem with the standard distro of amq and camel (As built from svn) by
>  just adding a single route (with activemq: as the component) and exercising
>  it.

Great - let me take a look...
--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 20/03/2008, James Strachan <[hidden email]> wrote:

> On 20/03/2008, DominicTulley <[hidden email]> wrote:
>  >
>
> >  Sorry for the noise - I just end up following my nose trying to find a way
>  >  round the problem.
>
>
> No problem :)
>
>
>  >  The easiest way to reproduce the problem is simply to pump a bunch of
>  >  messages through a standard camel route defined in activemq.xml.  I see the
>  >  problem with the standard distro of amq and camel (As built from svn) by
>  >  just adding a single route (with activemq: as the component) and exercising
>  >  it.
>
>
> Great - let me take a look...

So I've tried reproducing this and can't.

Here's what I did

* grabbed latest trunk of camel & built it
* grabbed latest trunk of activemq and built it

* ran the out of the box activemq tarball

cd activemq/assembly
tar xf apache-activemq*.tar.gz
cd apache-activemq*
bin/activemq

then ran the camel soak test in a separate JVM...

cd activemq/activemq-camel-loadtest
mvn test -Dtest=LocalBrokerLoadTest

This camel based load test will fire a bunch of messages into a queue
(example.A the same one used by default the camel route inside the
broker). Then it'll consume from (example.B) the one used by the camel
route in the broker - and assert it receives all the right messages in
order etc.

When I run this I see only 1 socket open connecting to port 61616
inside the JVM of the broker. I see only 1 socket connecting from the
other process.

e.g. running

 lsof -i | grep ":61616"

gave

java      13735 jstrachan   31u  IPv6 0x14670b2c      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:50651 (ESTABLISHED)
java      13735 jstrachan   33u  IPv6 0x1467fe4c      0t0  TCP
[::127.0.0.1]:50652->[::127.0.0.1]:61616 (ESTABLISHED)
java      13735 jstrachan   34u  IPv6 0x14682cd4      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:50652 (ESTABLISHED)
java      13735 jstrachan   48u  IPv6  0x6e97b2c      0t0  TCP *:61616 (LISTEN)
java      13735 jstrachan   65u  IPv6  0x6e960e0      0t0  TCP
[::127.0.0.1]:58826->[::127.0.0.1]:61616 (ESTABLISHED)
java      13735 jstrachan   66u  IPv6  0xd0a5be8      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:58826 (ESTABLISHED)
java      14100 jstrachan    7u  IPv6 0x146a319c      0t0  TCP
[::127.0.0.1]:50564->[::127.0.0.1]:61616 (ESTABLISHED)

There's a bunch of other sockets on different ports in the broker; as
the broker by default starts Jetty with the web console & web demo
along with other transports (SSL / stomp) and JMX etc.


I wonder if you try the above steps, do you get the same results?

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
Hi James,

I've run this test and I still get many time_Wait sockets (in "both" directions):

  TCP    ukedfinger:4996        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4997        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4998        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4999        localhost:61616        TIME_WAIT
  TCP    ukedfinger:61616       localhost:1540         TIME_WAIT
  TCP    ukedfinger:61616       localhost:1548         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2832         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2878         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2886         TIME_WAIT


I see from your message that you appear to be using unix/linux.  Perhaps I neglected to mention that I'm on windows?  I've been attempting to repeat your experiment on solaris but I'm struggling to get a subversion client so far - I will keep trying.

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
I wasn't able to run quite the test you specified on solaris (no svn or mvn installed), but I ran the tests I've been using with the broker on a solaris 10 machine and both a producer and a consumer on a windows machine.

I get the same results - hundreds of TIME_WAIT sockets (all localhost -> localhost) on the broker.

It's really strange that you don't see this!

What OS are you trying on?  Have you modified the tcp_time_wait_interval to be really short?

-Dominic


DominicTulley wrote
Hi James,

I've run this test and I still get many time_Wait sockets (in "both" directions):

  TCP    ukedfinger:4996        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4997        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4998        localhost:61616        TIME_WAIT
  TCP    ukedfinger:4999        localhost:61616        TIME_WAIT
  TCP    ukedfinger:61616       localhost:1540         TIME_WAIT
  TCP    ukedfinger:61616       localhost:1548         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2832         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2878         TIME_WAIT
  TCP    ukedfinger:61616       localhost:2886         TIME_WAIT


I see from your message that you appear to be using unix/linux.  Perhaps I neglected to mention that I'm on windows?  I've been attempting to repeat your experiment on solaris but I'm struggling to get a subversion client so far - I will keep trying.

-Dominic
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

jstrachan
On 24/03/2008, DominicTulley <[hidden email]> wrote:

>
>  I wasn't able to run quite the test you specified on solaris (no svn or mvn
>  installed), but I ran the tests I've been using with the broker on a solaris
>  10 machine and both a producer and a consumer on a windows machine.
>
>  I get the same results - hundreds of TIME_WAIT sockets (all localhost ->
>  localhost) on the broker.
>
>  It's really strange that you don't see this!
>
>  What OS are you trying on?  Have you modified the tcp_time_wait_interval to
>  be really short?

I tried OS X and Linux. e.g. here's the output on linux mid-way
through the test run when a few hundred messages had been consumed...

$  lsof -i | grep ":61616"
java    25789 jstrachan   20u  IPv6 1820489       TCP
localhost:38073->localhost:61616 (ESTABLISHED)
java    25789 jstrachan   21u  IPv6 1819230       TCP
localhost:61616->localhost:37506 (ESTABLISHED)
java    25789 jstrachan   26u  IPv6 1818559       TCP *:61616 (LISTEN)
java    25789 jstrachan   42u  IPv6 1818581       TCP
localhost:55527->localhost:61616 (ESTABLISHED)
java    25789 jstrachan   43u  IPv6 1818582       TCP
localhost:61616->localhost:55527 (ESTABLISHED)
java    25789 jstrachan   44u  IPv6 1820488       TCP
localhost:61616->localhost:38072 (ESTABLISHED)
java    25789 jstrachan   47u  IPv6 1820490       TCP
localhost:61616->localhost:38073 (ESTABLISHED)
java    25969 jstrachan    6u  IPv6 1820487       TCP
localhost:38072->localhost:61616 (ESTABLISHED)
java    25969 jstrachan    8u  IPv6 1819228       TCP
localhost:37506->localhost:61616 (ESTABLISHED)



The bizarre thing is - the code should only create a single socket &
never create another one - IF - the CACHE_CONSUMER setting is enabled.

I wonder if its worth turning on debug logging and sending the broker
log? I'm wondering if on your windows box sockets are failing alot? Or
maybe its the multicast notifications in the broker causing lots of
sockets to be opened? (Is it worth disabling the multicast discovery?)

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com
Reply | Threaded
Open this post in threaded view
|

Re: Getting lots of TIME_WAIT sockets

DominicTulley
In reply to this post by jstrachan
Hi James, I've just realised..

lsof -i doesn't show TIME_WAIT sockets!

> Great - let me take a look...

So I've tried reproducing this and can't.

Here's what I did

When I run this I see only 1 socket open connecting to port 61616
inside the JVM of the broker. I see only 1 socket connecting from the
other process.

e.g. running

 lsof -i | grep ":61616"

gave

java      13735 jstrachan   31u  IPv6 0x14670b2c      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:50651 (ESTABLISHED)
java      13735 jstrachan   33u  IPv6 0x1467fe4c      0t0  TCP
[::127.0.0.1]:50652->[::127.0.0.1]:61616 (ESTABLISHED)
java      13735 jstrachan   34u  IPv6 0x14682cd4      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:50652 (ESTABLISHED)
java      13735 jstrachan   48u  IPv6  0x6e97b2c      0t0  TCP *:61616 (LISTEN)
java      13735 jstrachan   65u  IPv6  0x6e960e0      0t0  TCP
[::127.0.0.1]:58826->[::127.0.0.1]:61616 (ESTABLISHED)
java      13735 jstrachan   66u  IPv6  0xd0a5be8      0t0  TCP
[::127.0.0.1]:61616->[::127.0.0.1]:58826 (ESTABLISHED)
java      14100 jstrachan    7u  IPv6 0x146a319c      0t0  TCP
[::127.0.0.1]:50564->[::127.0.0.1]:61616 (ESTABLISHED)

There's a bunch of other sockets on different ports in the broker; as
the broker by default starts Jetty with the web console & web demo
along with other transports (SSL / stomp) and JMX etc.


I wonder if you try the above steps, do you get the same results?

--
James
-------
http://macstrac.blogspot.com/

Open Source Integration
http://open.iona.com


12