Possible memory leak when using Camel in ActiveMQ

[DominicTulley]

I've been doing some throughput experiments to assess the stability of Camel and I have seen a worrying memory usage pattern (see attached screen shot graph1.JPG).

This ran overnight and processed around 900,000 messages through a fairly simple router (it was doing a hashmap lookup and setting a header on each message).

By the time the run ended, there were several thousand backed up messages in the queue (camel was not processing as fast as the messages were arriving).  The flat part at the end of the graph is when the consumer was left running to remove those messages.  The system was given every opportunity to garbage collect (and had done many thousands of times of course).

So far, I cannot prove that there is a memory leak.  Certainly this memory usage pattern does not occur if I just pump messages through ActiveMQ (heap usage stays at a constant 10Mb or so).  I have been attempting to profile this to find the culprit but it all runs so slowly I'm struggling to get any conclusive results.

So, I guess I'm asking if
1)  This is an expected memory usage profile and eventually stuff does get discarded
2)  Anyone knows of a problem in this area
3)  Anyone has got any clever ideas for how to profile this more successfully

Thanks,

-Dominic

[Hiram Chirino]

What route were you using?  That will us focus and the possible leaks better.

On 8/29/07, DominicTulley <[hidden email]> wrote:

>
> I've been doing some throughput experiments to assess the stability of Camel
> and I have seen a worrying memory usage pattern (see attached screen shot
> http://www.nabble.com/file/p12384346/graph1.JPG graph1.JPG ).
>
> This ran overnight and processed around 900,000 messages through a fairly
> simple router (it was doing a hashmap lookup and setting a header on each
> message).
>
> By the time the run ended, there were several thousand backed up messages in
> the queue (camel was not processing as fast as the messages were arriving).
> The flat part at the end of the graph is when the consumer was left running
> to remove those messages.  The system was given every opportunity to garbage
> collect (and had done many thousands of times of course).
>
> So far, I cannot prove that there is a memory leak.  Certainly this memory
> usage pattern does not occur if I just pump messages through ActiveMQ (heap
> usage stays at a constant 10Mb or so).  I have been attempting to profile
> this to find the culprit but it all runs so slowly I'm struggling to get any
> conclusive results.
>
> So, I guess I'm asking if
> 1)  This is an expected memory usage profile and eventually stuff does get
> discarded
> 2)  Anyone knows of a problem in this area
> 3)  Anyone has got any clever ideas for how to profile this more
> successfully
>
> Thanks,
>
> -Dominic
> --
> View this message in context: http://www.nabble.com/Possible-memory-leak-when-using-Camel-in-ActiveMQ-tf4346921s22882.html#a12384346
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
>

--
Regards,
Hiram

Blog: http://hiramchirino.com

[DominicTulley]

The route I am currently using is:

from("activemq:AnnotateQueue").setHeader("HasHandler", hasHandlerExpression).to("activemq:TOOL.DEFAULT");;

where hasHandlerExpression is a class I have implemented.  It contains the following method:

        public Object evaluate(Exchange exch) {
                Message msg = exch.getIn();
                String key = (String)msg.getHeader("DocGuid");
               
                // Odd numbered docs have a handler
                 
                String handlerStr = RoutingTable.getDestination(key);
                boolean hasHandler = (null != handlerStr) && handlerStr.equals("true");
               
                msg.setHeader("HasHandler", hasHandler);
                return "" + hasHandler;

Finally RoutingTable.getDestination is simply a static method doing a hashmap.get.

Hope this helps.

-Dominic


<quote author="Hiram Chirino">
What route were you using?  That will us focus and the possible leaks better.

[Nicky Sandhu]

DominicTulley wrote

So far, I cannot prove that there is a memory leak.  Certainly this memory usage pattern does not occur if I just pump messages through ActiveMQ (heap usage stays at a constant 10Mb or so).  I have been attempting to profile this to find the culprit but it all runs so slowly I'm struggling to get any conclusive results.

So, I guess I'm asking if
1)  This is an expected memory usage profile and eventually stuff does get discarded
2)  Anyone knows of a problem in this area
3)  Anyone has got any clever ideas for how to profile this more successfully

Memory leaks are easier to diagnose in JDK6 using jhat and jmap. I have used it in the past to pinpoint which object is holding on to what and making an educated guess based on that.
Here's a blog that shows how to do it http://weblogs.java.net/blog/jfarcand/archive/2006/02/using_mustangs.html.
Here's another for general troubleshooting links
http://java.sun.com/javase/6/webnotes/trouble/