OOM issue due to MemoryIdempotentRepository

classic Classic list List threaded Threaded
18 messages Options
Reply | Threaded
Open this post in threaded view
|

OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi All,

I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.


Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.


In my heap dump I can see the following trace as the major memory consumers:

java.util.ArrayList
 >java.lang.Object

    >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService

        >org.apache.camel.component.file.FileEndpoint

            >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
                 >org.apache.camel.util.LRUCache
                    >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap


So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?

I am testing this scenario under Storm as execution environment and with 2GB max heap space.

Thanks in advance

Chirag Dewan
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Claus Ibsen-2
Hi

Some days ago someone else posted about a OOME issue when using Storm.
It smells like a Storm + Camel issue somewhere.


On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>
>
> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>
>
> In my heap dump I can see the following trace as the major memory consumers:
>
> java.util.ArrayList
>  >java.lang.Object
>
>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>
>         >org.apache.camel.component.file.FileEndpoint
>
>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>                  >org.apache.camel.util.LRUCache
>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
>
> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>
> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>
> Thanks in advance
>
> Chirag Dewan



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi Claus,

Thanks for the quick reply.

It might well be. But there is one thing I would like to get some insight into.

If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.

org.apache.camel.util.LRUCache
  >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap

I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?

Thanks.

Chirag Dewan





________________________________
 From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 5:42 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi

Some days ago someone else posted about a OOME issue when using Storm.
It smells like a Storm + Camel issue somewhere.



On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>
>
> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>
>
> In my heap dump I can see the following trace as the major memory consumers:
>
> java.util.ArrayList
>  >java.lang.Object
>
>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>
>         >org.apache.camel.component.file.FileEndpoint
>
>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>                  >org.apache.camel.util.LRUCache
>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
>
> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>
> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>
> Thanks in advance
>
> Chirag Dewan



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi,

And I am using readLock=changed. So that can be a reason too?

Chirag Dewan




________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 6:07 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi Claus,

Thanks for the quick reply.

It might well be. But there is one thing I would like to get some insight into.

If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.

org.apache.camel.util.LRUCache
  >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap

I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?

Thanks.

Chirag Dewan





________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 5:42 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi

Some days ago someone else posted about a OOME issue when using Storm.
It smells like a Storm + Camel issue somewhere.



On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>
>
> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>
>
> In my heap dump I can see the following trace as the major memory consumers:
>
> java.util.ArrayList
>  >java.lang.Object
>
>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>
>         >org.apache.camel.component.file.FileEndpoint
>
>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>                  >org.apache.camel.util.LRUCache
>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
>
> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>
> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>
> Thanks in advance
>
> Chirag Dewan



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi All,

I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?

Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?

Any help is much appreciated.

Thanks!

Chirag Dewan




________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
Sent: Wednesday, 12 February 2014 6:31 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi,

And I am using readLock=changed. So that can be a reason too?

Chirag Dewan




________________________________

From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 6:07 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi Claus,

Thanks for the quick reply.

It might well be. But there is one thing I would like to get some insight into.

If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.

org.apache.camel.util.LRUCache
  >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap

I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?

Thanks.

Chirag Dewan





________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 5:42 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi

Some days ago someone else posted about a OOME issue when using Storm.
It smells like a Storm + Camel issue somewhere.



On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>
>
> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>
>
> In my heap dump I can see the following trace as the major memory consumers:
>
> java.util.ArrayList
>  >java.lang.Object
>
>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>
>         >org.apache.camel.component.file.FileEndpoint
>
>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>                  >org.apache.camel.util.LRUCache
>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
>
> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>
> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>
> Thanks in advance
>
> Chirag Dewan



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi All,

Any views on this?

Actually my previous mail is not understandable. Let me rephrase the issue.

I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.

I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.

Is there something I am missing here?

Thanks in advance.

Chirag Dewan




________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
Sent: Friday, 14 February 2014 3:22 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi All,

I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?

Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?

Any help is much appreciated.

Thanks!

Chirag Dewan




________________________________

From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
Sent: Wednesday, 12 February 2014 6:31 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi,

And I am using readLock=changed. So that can be a reason too?

Chirag Dewan




________________________________

From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 6:07 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi Claus,

Thanks for the quick reply.

It might well be. But there is one thing I would like to get some insight into.

If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.

org.apache.camel.util.LRUCache
  >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap

I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?

Thanks.

Chirag Dewan





________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 12 February 2014 5:42 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi

Some days ago someone else posted about a OOME issue when using Storm.
It smells like a Storm + Camel issue somewhere.



On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>
>
> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>
>
> In my heap dump I can see the following trace as the major memory consumers:
>
> java.util.ArrayList
>  >java.lang.Object
>
>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>
>         >org.apache.camel.component.file.FileEndpoint
>
>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>                  >org.apache.camel.util.LRUCache
>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
>
> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>
> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>
> Thanks in advance
>
> Chirag Dewan



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Claus Ibsen-2
Hi

You should likely reuse the consumer template instead. As this FAQ
applies to it too
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html

On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> Any views on this?
>
> Actually my previous mail is not understandable. Let me rephrase the issue.
>
> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>
> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>
> Is there something I am missing here?
>
> Thanks in advance.
>
> Chirag Dewan
>
>
>
>
> ________________________________
>  From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Friday, 14 February 2014 3:22 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi All,
>
> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>
> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>
> Any help is much appreciated.
>
> Thanks!
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Wednesday, 12 February 2014 6:31 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi,
>
> And I am using readLock=changed. So that can be a reason too?
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 12 February 2014 6:07 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> Thanks for the quick reply.
>
> It might well be. But there is one thing I would like to get some insight into.
>
> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>
> org.apache.camel.util.LRUCache
>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 12 February 2014 5:42 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> Some days ago someone else posted about a OOME issue when using Storm.
> It smells like a Storm + Camel issue somewhere.
>
>
>
> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi All,
>>
>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>
>>
>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>
>>
>> In my heap dump I can see the following trace as the major memory consumers:
>>
>> java.util.ArrayList
>>  >java.lang.Object
>>
>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>
>>         >org.apache.camel.component.file.FileEndpoint
>>
>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>                  >org.apache.camel.util.LRUCache
>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>>
>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>
>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>
>> Thanks in advance
>>
>> Chirag Dewan
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi Claus,

So I dont need to start/stop the template for every file?

Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.

Is this the right approach?

Thanks.

Chirag




________________________________
 From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Monday, 17 February 2014 12:33 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi

You should likely reuse the consumer template instead. As this FAQ
applies to it too
http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html

On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:

> Hi All,
>
> Any views on this?
>
> Actually my previous mail is not understandable. Let me rephrase the issue.
>
> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>
> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>
> Is there something I am missing here?
>
> Thanks in advance.
>
> Chirag Dewan
>
>
>
>
> ________________________________
>  From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Friday, 14 February 2014 3:22 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi All,
>
> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>
> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>
> Any help is much appreciated.
>
> Thanks!
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Wednesday, 12 February 2014 6:31 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi,
>
> And I am using readLock=changed. So that can be a reason too?
>
> Chirag Dewan
>
>
>
>
> ________________________________
>
> From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 12 February 2014 6:07 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> Thanks for the quick reply.
>
> It might well be. But there is one thing I would like to get some insight into.
>
> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>
> org.apache.camel.util.LRUCache
>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>
> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 12 February 2014 5:42 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> Some days ago someone else posted about a OOME issue when using Storm.
> It smells like a Storm + Camel issue somewhere.
>
>
>
> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi All,
>>
>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>
>>
>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>
>>
>> In my heap dump I can see the following trace as the major memory consumers:
>>
>> java.util.ArrayList
>>  >java.lang.Object
>>
>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>
>>         >org.apache.camel.component.file.FileEndpoint
>>
>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>                  >org.apache.camel.util.LRUCache
>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>>
>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>
>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>
>> Thanks in advance
>>
>> Chirag Dewan
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io




--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Claus Ibsen-2
On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
> Hi Claus,
>
> So I dont need to start/stop the template for every file?
>
> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>
> Is this the right approach?
>

No, only start|stop the template once on startup and shutdown of your app.
And then call doneUoW for each file you consume.


> Thanks.
>
> Chirag
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 12:33 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> You should likely reuse the consumer template instead. As this FAQ
> applies to it too
> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>
> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi All,
>>
>> Any views on this?
>>
>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>
>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>
>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>
>> Is there something I am missing here?
>>
>> Thanks in advance.
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Friday, 14 February 2014 3:22 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi All,
>>
>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>
>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>
>> Any help is much appreciated.
>>
>> Thanks!
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:31 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi,
>>
>> And I am using readLock=changed. So that can be a reason too?
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:07 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> Thanks for the quick reply.
>>
>> It might well be. But there is one thing I would like to get some insight into.
>>
>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>
>> org.apache.camel.util.LRUCache
>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>
>> Thanks.
>>
>> Chirag Dewan
>>
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 5:42 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> Some days ago someone else posted about a OOME issue when using Storm.
>> It smells like a Storm + Camel issue somewhere.
>>
>>
>>
>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>
>>>
>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>
>>>
>>> In my heap dump I can see the following trace as the major memory consumers:
>>>
>>> java.util.ArrayList
>>>  >java.lang.Object
>>>
>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>
>>>         >org.apache.camel.component.file.FileEndpoint
>>>
>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>                  >org.apache.camel.util.LRUCache
>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>>
>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>
>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>
>>> Thanks in advance
>>>
>>> Chirag Dewan
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi Claus,

I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?

Thanks .

Chirag 




________________________________
 From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
Sent: Monday, 17 February 2014 1:28 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
> Hi Claus,
>
> So I dont need to start/stop the template for every file?
>
> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>
> Is this the right approach?
>

No, only start|stop the template once on startup and shutdown of your app.
And then call doneUoW for each file you consume.


> Thanks.
>
> Chirag
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 12:33 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> You should likely reuse the consumer template instead. As this FAQ
> applies to it too
> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>
> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi All,
>>
>> Any views on this?
>>
>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>
>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>
>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>
>> Is there something I am missing here?
>>
>> Thanks in advance.
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Friday, 14 February 2014 3:22 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi All,
>>
>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>
>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>
>> Any help is much appreciated.
>>
>> Thanks!
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:31 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi,
>>
>> And I am using readLock=changed. So that can be a reason too?
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:07 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> Thanks for the quick reply.
>>
>> It might well be. But there is one thing I would like to get some insight into.
>>
>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>
>> org.apache.camel.util.LRUCache
>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>
>> Thanks.
>>
>> Chirag Dewan
>>
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 5:42 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> Some days ago someone else posted about a OOME issue when using Storm.
>> It smells like a Storm + Camel issue somewhere.
>>
>>
>>
>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>
>>>
>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>
>>>
>>> In my heap dump I can see the following trace as the major memory consumers:
>>>
>>> java.util.ArrayList
>>>  >java.lang.Object
>>>
>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>
>>>         >org.apache.camel.component.file.FileEndpoint
>>>
>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>                  >org.apache.camel.util.LRUCache
>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>>
>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>
>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>
>>> Thanks in advance
>>>
>>> Chirag Dewan
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io

>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan


Hi Claus,

I had this implementation
before. With this I saw an unusual behaviour. A thread was created for
every file consumed by the template and even though I called
template.doneUoW the thread is not stopped. So if I call doneUoW(and not
 stop) the thread should be destroyed? And also on doneUoW the
inprogressRepository cache is cleared. Which is not getting cleared at
the moment(the reason I believe is causing the OOM) . Is there a possibility
that doneUoW is not working as it should?

Thanks .

Chirag 

________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Monday, 17 February 2014 1:53 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi Claus,

I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?

Thanks .

Chirag 




________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
Sent: Monday, 17 February 2014 1:28 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
> Hi Claus,
>
> So I dont need to start/stop the template for every file?
>
> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>
> Is this the right approach?
>

No, only start|stop the template once on startup and shutdown of your app.
And then call doneUoW for each file you consume.


> Thanks.
>
> Chirag
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 12:33 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> You should likely reuse the consumer template instead. As this FAQ
> applies to it too
> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>
> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi All,
>>
>> Any views on this?
>>
>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>
>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>
>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>
>> Is there something I am missing here?
>>
>> Thanks in advance.
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Friday, 14 February 2014 3:22 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi All,
>>
>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>
>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>
>> Any help is much appreciated.
>>
>> Thanks!
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:31 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi,
>>
>> And I am using readLock=changed. So that can be a reason too?
>>
>> Chirag Dewan
>>
>>
>>
>>
>> ________________________________
>>
>> From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 6:07 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> Thanks for the quick reply.
>>
>> It might well be. But there is one thing I would like to get some insight into.
>>
>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>
>> org.apache.camel.util.LRUCache
>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>
>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>
>> Thanks.
>>
>> Chirag Dewan
>>
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Wednesday, 12 February 2014 5:42 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> Some days ago someone else posted about a OOME issue when using Storm.
>> It smells like a Storm + Camel issue somewhere.
>>
>>
>>
>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>
>>>
>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>
>>>
>>> In my heap dump I can see the following trace as the major memory consumers:
>>>
>>> java.util.ArrayList
>>>  >java.lang.Object
>>>
>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>
>>>         >org.apache.camel.component.file.FileEndpoint
>>>
>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>                  >org.apache.camel.util.LRUCache
>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>>
>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>
>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>
>>> Thanks in advance
>>>
>>> Chirag Dewan
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io

>
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Claus Ibsen-2
Hi

What is the exact url you use in the consumer template?
And are you 100% sure you only create and use exactly one instance of
the consumer template?

On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:

>
>
> Hi Claus,
>
> I had this implementation
> before. With this I saw an unusual behaviour. A thread was created for
> every file consumed by the template and even though I called
> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>  stop) the thread should be destroyed? And also on doneUoW the
> inprogressRepository cache is cleared. Which is not getting cleared at
> the moment(the reason I believe is causing the OOM) . Is there a possibility
> that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
> ________________________________
>  From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 1:53 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Monday, 17 February 2014 1:28 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi Claus,
>>
>> So I dont need to start/stop the template for every file?
>>
>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>
>> Is this the right approach?
>>
>
> No, only start|stop the template once on startup and shutdown of your app.
> And then call doneUoW for each file you consume.
>
>
>> Thanks.
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>  From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 12:33 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> You should likely reuse the consumer template instead. As this FAQ
>> applies to it too
>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>
>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> Any views on this?
>>>
>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>
>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>
>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>
>>> Is there something I am missing here?
>>>
>>> Thanks in advance.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Friday, 14 February 2014 3:22 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi All,
>>>
>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>
>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi,
>>>
>>> And I am using readLock=changed. So that can be a reason too?
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi Claus,
>>>
>>> Thanks for the quick reply.
>>>
>>> It might well be. But there is one thing I would like to get some insight into.
>>>
>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>
>>> org.apache.camel.util.LRUCache
>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>
>>> Thanks.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> Some days ago someone else posted about a OOME issue when using Storm.
>>> It smells like a Storm + Camel issue somewhere.
>>>
>>>
>>>
>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>
>>>>
>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>
>>>>
>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>
>>>> java.util.ArrayList
>>>>  >java.lang.Object
>>>>
>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>
>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>
>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>                  >org.apache.camel.util.LRUCache
>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>>
>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>
>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>
>>>> Thanks in advance
>>>>
>>>> Chirag Dewan
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io
>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

skasish
In reply to this post by Chirag Dewan
Chirag,

I have got the same problem.

Have you got the solution.
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
In reply to this post by Claus Ibsen-2
Hi Claus,

Sorry for the late reply.

I wrote a simple code to test this behaviour:

        CamelContext con = new DefaultCamelContext();

        ConsumerTemplate template = con.createConsumerTemplate();

        template.start();

        int i = 0;

        while (i < 100) {


            Exchange exchange = template.receive(
                    "file:/sourceDir?fileName=abc (" + i
                            + "th copy).txt&delete=true", 10000);

            System.out.println(exchange + " " + i);
           

             template.doneUoW(exchange);

            i++;

          

        }
       
        template.stop();

Even on profiling this I see a thread being created for every file consumed by the template. And the memory continous to grow.

Thanks.

Chirag Dewan

      




________________________________
 From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 19 February 2014 2:59 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi

What is the exact url you use in the consumer template?
And are you 100% sure you only create and use exactly one instance of
the consumer template?

On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:

>
>
> Hi Claus,
>
> I had this implementation
> before. With this I saw an unusual behaviour. A thread was created for
> every file consumed by the template and even though I called
> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>  stop) the thread should be destroyed? And also on doneUoW the
> inprogressRepository cache is cleared. Which is not getting cleared at
> the moment(the reason I believe is causing the OOM) . Is there a possibility
> that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
> ________________________________
>  From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 1:53 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Monday, 17 February 2014 1:28 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi Claus,
>>
>> So I dont need to start/stop the template for every file?
>>
>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>
>> Is this the right approach?
>>
>
> No, only start|stop the template once on startup and shutdown of your app.
> And then call doneUoW for each file you consume.
>
>
>> Thanks.
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>  From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 12:33 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> You should likely reuse the consumer template instead. As this FAQ
>> applies to it too
>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>
>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> Any views on this?
>>>
>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>
>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>
>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>
>>> Is there something I am missing here?
>>>
>>> Thanks in advance.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Friday, 14 February 2014 3:22 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi All,
>>>
>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>
>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi,
>>>
>>> And I am using readLock=changed. So that can be a reason too?
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi Claus,
>>>
>>> Thanks for the quick reply.
>>>
>>> It might well be. But there is one thing I would like to get some insight into.
>>>
>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>
>>> org.apache.camel.util.LRUCache
>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>
>>> Thanks.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> Some days ago someone else posted about a OOME issue when using Storm.
>>> It smells like a Storm + Camel issue somewhere.
>>>
>>>
>>>
>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>
>>>>
>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>
>>>>
>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>
>>>> java.util.ArrayList
>>>>  >java.lang.Object
>>>>
>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>
>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>
>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>                  >org.apache.camel.util.LRUCache
>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>>
>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>
>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>
>>>> Thanks in advance
>>>>
>>>> Chirag Dewan
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io

>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
And when I call template.stop(),all the threads are destroyed.




________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Thursday, 20 February 2014 3:23 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi Claus,

Sorry for the late reply.

I wrote a simple code to test this behaviour:

        CamelContext con = new DefaultCamelContext();

        ConsumerTemplate template = con.createConsumerTemplate();

        template.start();

        int i = 0;

        while (i < 100) {


            Exchange exchange = template.receive(
                    "file:/sourceDir?fileName=abc (" + i
                            + "th copy).txt&delete=true", 10000);

            System.out.println(exchange + " " + i);
           

             template.doneUoW(exchange);

            i++;

          

        }
       
        template.stop();

Even on profiling this I see a thread being created for every file consumed by the template. And the memory continous to grow.

Thanks.

Chirag Dewan

      




________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Wednesday, 19 February 2014 2:59 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi

What is the exact url you use in the consumer template?
And are you 100% sure you only create and use exactly one instance of
the consumer template?

On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:

>
>
> Hi Claus,
>
> I had this implementation
> before. With this I saw an unusual behaviour. A thread was created for
> every file consumed by the template and even though I called
> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>  stop) the thread should be destroyed? And also on doneUoW the
> inprogressRepository cache is cleared. Which is not getting cleared at
> the moment(the reason I believe is causing the OOM) . Is there a possibility
> that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
> ________________________________
>  From: Chirag Dewan <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Monday, 17 February 2014 1:53 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi Claus,
>
> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>
> Thanks .
>
> Chirag
>
>
>
>
> ________________________________
>
> From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
> Sent: Monday, 17 February 2014 1:28 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>> Hi Claus,
>>
>> So I dont need to start/stop the template for every file?
>>
>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>
>> Is this the right approach?
>>
>
> No, only start|stop the template once on startup and shutdown of your app.
> And then call doneUoW for each file you consume.
>
>
>> Thanks.
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>  From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 12:33 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi
>>
>> You should likely reuse the consumer template instead. As this FAQ
>> applies to it too
>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>
>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi All,
>>>
>>> Any views on this?
>>>
>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>
>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>
>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>
>>> Is there something I am missing here?
>>>
>>> Thanks in advance.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Friday, 14 February 2014 3:22 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi All,
>>>
>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>
>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>
>>> Any help is much appreciated.
>>>
>>> Thanks!
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi,
>>>
>>> And I am using readLock=changed. So that can be a reason too?
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Chirag Dewan <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi Claus,
>>>
>>> Thanks for the quick reply.
>>>
>>> It might well be. But there is one thing I would like to get some insight into.
>>>
>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>
>>> org.apache.camel.util.LRUCache
>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>
>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>
>>> Thanks.
>>>
>>> Chirag Dewan
>>>
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>> From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> Some days ago someone else posted about a OOME issue when using Storm.
>>> It smells like a Storm + Camel issue somewhere.
>>>
>>>
>>>
>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>
>>>>
>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>
>>>>
>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>
>>>> java.util.ArrayList
>>>>  >java.lang.Object
>>>>
>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>
>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>
>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>                  >org.apache.camel.util.LRUCache
>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>>
>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>
>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>
>>>> Thanks in advance
>>>>
>>>> Chirag Dewan
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io

>
>>
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Claus Ibsen-2
In reply to this post by Chirag Dewan
Hi

That is because the endpoint uri, is unique, eg you put the index in there.
Then the endpoint cannot be reused for next poll.

See this FAQ how you can lower the pool size
http://camel.apache.org/how-do-i-configure-the-default-maximum-cache-size-for-producercache-or-producertemplate.html

You can set it to 1 or 0 to not have any pooling.

On Thu, Feb 20, 2014 at 10:53 AM, Chirag Dewan <[hidden email]> wrote:

> Hi Claus,
>
> Sorry for the late reply.
>
> I wrote a simple code to test this behaviour:
>
>         CamelContext con = new DefaultCamelContext();
>
>         ConsumerTemplate template = con.createConsumerTemplate();
>
>         template.start();
>
>         int i = 0;
>
>         while (i < 100) {
>
>
>             Exchange exchange = template.receive(
>                     "file:/sourceDir?fileName=abc (" + i
>                             + "th copy).txt&delete=true", 10000);
>
>             System.out.println(exchange + " " + i);
>
>
>              template.doneUoW(exchange);
>
>             i++;
>
>
>
>         }
>
>         template.stop();
>
> Even on profiling this I see a thread being created for every file consumed by the template. And the memory continous to grow.
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 19 February 2014 2:59 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> What is the exact url you use in the consumer template?
> And are you 100% sure you only create and use exactly one instance of
> the consumer template?
>
> On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:
>>
>>
>> Hi Claus,
>>
>> I had this implementation
>> before. With this I saw an unusual behaviour. A thread was created for
>> every file consumed by the template and even though I called
>> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>>  stop) the thread should be destroyed? And also on doneUoW the
>> inprogressRepository cache is cleared. Which is not getting cleared at
>> the moment(the reason I believe is causing the OOM) . Is there a possibility
>> that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 1:53 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Monday, 17 February 2014 1:28 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi Claus,
>>>
>>> So I dont need to start/stop the template for every file?
>>>
>>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>>
>>> Is this the right approach?
>>>
>>
>> No, only start|stop the template once on startup and shutdown of your app.
>> And then call doneUoW for each file you consume.
>>
>>
>>> Thanks.
>>>
>>> Chirag
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Monday, 17 February 2014 12:33 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> You should likely reuse the consumer template instead. As this FAQ
>>> applies to it too
>>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>>
>>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> Any views on this?
>>>>
>>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>>
>>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>>
>>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>>
>>>> Is there something I am missing here?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>  From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Friday, 14 February 2014 3:22 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>>
>>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>>
>>>> Any help is much appreciated.
>>>>
>>>> Thanks!
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi,
>>>>
>>>> And I am using readLock=changed. So that can be a reason too?
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi Claus,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> It might well be. But there is one thing I would like to get some insight into.
>>>>
>>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>>
>>>> org.apache.camel.util.LRUCache
>>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>>
>>>> Thanks.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Claus Ibsen <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi
>>>>
>>>> Some days ago someone else posted about a OOME issue when using Storm.
>>>> It smells like a Storm + Camel issue somewhere.
>>>>
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>>
>>>>>
>>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>>
>>>>>
>>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>>
>>>>> java.util.ArrayList
>>>>>  >java.lang.Object
>>>>>
>>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>>
>>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>>
>>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>>                  >org.apache.camel.util.LRUCache
>>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>>
>>>>>
>>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>>
>>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> Red Hat, Inc.
>>>> Email: [hidden email]
>>>> Twitter: davsclaus
>>>> Blog: http://davsclaus.com
>>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>> Make your Camel applications look hawt, try: http://hawt.io
>
>>
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi,

After setting the max cache for consumer template the threads are now in WAITING state instead of TIMED_WAITING state.

Actually I want to consume a single file with a filename in my application. So once a file is consumed,that endpoint cannot be reused and for every file a new endpoint is created. Is there a better way of doing this? 

And also the memory consumed by MemoryIdempotentRepository,could this be the reason for that?

Thanks


Chirag




________________________________
 From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Thursday, 20 February 2014 3:46 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi

That is because the endpoint uri, is unique, eg you put the index in there.
Then the endpoint cannot be reused for next poll.

See this FAQ how you can lower the pool size
http://camel.apache.org/how-do-i-configure-the-default-maximum-cache-size-for-producercache-or-producertemplate.html

You can set it to 1 or 0 to not have any pooling.

On Thu, Feb 20, 2014 at 10:53 AM, Chirag Dewan <[hidden email]> wrote:

> Hi Claus,
>
> Sorry for the late reply.
>
> I wrote a simple code to test this behaviour:
>
>         CamelContext con = new DefaultCamelContext();
>
>         ConsumerTemplate template = con.createConsumerTemplate();
>
>         template.start();
>
>         int i = 0;
>
>         while (i < 100) {
>
>
>             Exchange exchange = template.receive(
>                     "file:/sourceDir?fileName=abc (" + i
>                             + "th copy).txt&delete=true", 10000);
>
>             System.out.println(exchange + " " + i);
>
>
>              template.doneUoW(exchange);
>
>             i++;
>
>
>
>         }
>
>         template.stop();
>
> Even on profiling this I see a thread being created for every file consumed by the template. And the memory continous to grow.
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 19 February 2014 2:59 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> What is the exact url you use in the consumer template?
> And are you 100% sure you only create and use exactly one instance of
> the consumer template?
>
> On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:
>>
>>
>> Hi Claus,
>>
>> I had this implementation
>> before. With this I saw an unusual behaviour. A thread was created for
>> every file consumed by the template and even though I called
>> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>>  stop) the thread should be destroyed? And also on doneUoW the
>> inprogressRepository cache is cleared. Which is not getting cleared at
>> the moment(the reason I believe is causing the OOM) . Is there a possibility
>> that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 1:53 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Monday, 17 February 2014 1:28 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi Claus,
>>>
>>> So I dont need to start/stop the template for every file?
>>>
>>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>>
>>> Is this the right approach?
>>>
>>
>> No, only start|stop the template once on startup and shutdown of your app.
>> And then call doneUoW for each file you consume.
>>
>>
>>> Thanks.
>>>
>>> Chirag
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Monday, 17 February 2014 12:33 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> You should likely reuse the consumer template instead. As this FAQ
>>> applies to it too
>>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>>
>>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> Any views on this?
>>>>
>>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>>
>>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>>
>>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>>
>>>> Is there something I am missing here?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>  From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Friday, 14 February 2014 3:22 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>>
>>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>>
>>>> Any help is much appreciated.
>>>>
>>>> Thanks!
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi,
>>>>
>>>> And I am using readLock=changed. So that can be a reason too?
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi Claus,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> It might well be. But there is one thing I would like to get some insight into.
>>>>
>>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>>
>>>> org.apache.camel.util.LRUCache
>>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>>
>>>> Thanks.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Claus Ibsen <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi
>>>>
>>>> Some days ago someone else posted about a OOME issue when using Storm.
>>>> It smells like a Storm + Camel issue somewhere.
>>>>
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>>
>>>>>
>>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>>
>>>>>
>>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>>
>>>>> java.util.ArrayList
>>>>>  >java.lang.Object
>>>>>
>>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>>
>>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>>
>>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>>                  >org.apache.camel.util.LRUCache
>>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>>
>>>>>
>>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>>
>>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> Red Hat, Inc.
>>>> Email: [hidden email]
>>>> Twitter: davsclaus
>>>> Blog: http://davsclaus.com
>>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>> Make your Camel applications look hawt, try: http://hawt.io

>
>>
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io
Reply | Threaded
Open this post in threaded view
|

Re: OOM issue due to MemoryIdempotentRepository

Chirag Dewan
Hi,

I might be misusing the ConsumerTemplate. Given that its a polling consumer,and with File Endpoint it will keep polling for a directory(and not a single file) once started. My requirement is to consume a single file,and a large number of those too.

I might want to use the conventional Java file reading here.

Thanks a lot for you help.

Chirag Dewan



________________________________
 From: Chirag Dewan <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Thursday, 20 February 2014 4:06 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository
 

Hi,

After setting the max cache for consumer template the threads are now in WAITING state instead of TIMED_WAITING state.

Actually I want to consume a single file with a filename in my application. So once a file is consumed,that endpoint cannot be reused and for every file a new endpoint is created. Is there a better way of doing this? 

And also the memory consumed by MemoryIdempotentRepository,could this be the reason for that?

Thanks


Chirag




________________________________

From: Claus Ibsen <[hidden email]>
To: "[hidden email]" <[hidden email]>
Sent: Thursday, 20 February 2014 3:46 PM
Subject: Re: OOM issue due to MemoryIdempotentRepository


Hi

That is because the endpoint uri, is unique, eg you put the index in there.
Then the endpoint cannot be reused for next poll.

See this FAQ how you can lower the pool size
http://camel.apache.org/how-do-i-configure-the-default-maximum-cache-size-for-producercache-or-producertemplate.html

You can set it to 1 or 0 to not have any pooling.

On Thu, Feb 20, 2014 at 10:53 AM, Chirag Dewan <[hidden email]> wrote:

> Hi Claus,
>
> Sorry for the late reply.
>
> I wrote a simple code to test this behaviour:
>
>         CamelContext con = new DefaultCamelContext();
>
>         ConsumerTemplate template = con.createConsumerTemplate();
>
>         template.start();
>
>         int i = 0;
>
>         while (i < 100) {
>
>
>             Exchange exchange = template.receive(
>                     "file:/sourceDir?fileName=abc (" + i
>                             + "th copy).txt&delete=true", 10000);
>
>             System.out.println(exchange + " " + i);
>
>
>              template.doneUoW(exchange);
>
>             i++;
>
>
>
>         }
>
>         template.stop();
>
> Even on profiling this I see a thread being created for every file consumed by the template. And the memory continous to grow.
>
> Thanks.
>
> Chirag Dewan
>
>
>
>
>
>
> ________________________________
>  From: Claus Ibsen <[hidden email]>
> To: "[hidden email]" <[hidden email]>
> Sent: Wednesday, 19 February 2014 2:59 PM
> Subject: Re: OOM issue due to MemoryIdempotentRepository
>
>
> Hi
>
> What is the exact url you use in the consumer template?
> And are you 100% sure you only create and use exactly one instance of
> the consumer template?
>
> On Mon, Feb 17, 2014 at 9:26 AM, Chirag Dewan <[hidden email]> wrote:
>>
>>
>> Hi Claus,
>>
>> I had this implementation
>> before. With this I saw an unusual behaviour. A thread was created for
>> every file consumed by the template and even though I called
>> template.doneUoW the thread is not stopped. So if I call doneUoW(and not
>>  stop) the thread should be destroyed? And also on doneUoW the
>> inprogressRepository cache is cleared. Which is not getting cleared at
>> the moment(the reason I believe is causing the OOM) . Is there a possibility
>> that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>> ________________________________
>>  From: Chirag Dewan <[hidden email]>
>> To: "[hidden email]" <[hidden email]>
>> Sent: Monday, 17 February 2014 1:53 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> Hi Claus,
>>
>> I had this implementation before. With this I saw an unusual behaviour. A thread was created for every file consumed by the template and even though I called template.doneUoW the thread is not stopped. So if I call doneUoW(and not stop) the thread should be destroyed? And also on doneUoW the inprogressRepository cache is cleared. Which is not getting cleared at the moment(the reason I believe is causing the . Is there a possibility that doneUoW is not working as it should?
>>
>> Thanks .
>>
>> Chirag
>>
>>
>>
>>
>> ________________________________
>>
>> From: Claus Ibsen <[hidden email]>
>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>> Sent: Monday, 17 February 2014 1:28 PM
>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>
>>
>> On Mon, Feb 17, 2014 at 8:49 AM, Chirag Dewan <[hidden email]> wrote:
>>> Hi Claus,
>>>
>>> So I dont need to start/stop the template for every file?
>>>
>>> Right now I create a consumerTemplate at satrtup and then on every method call(responsible for consuming file) I start the template,consume the file(template.receive) and stop the template.
>>>
>>> Is this the right approach?
>>>
>>
>> No, only start|stop the template once on startup and shutdown of your app.
>> And then call doneUoW for each file you consume.
>>
>>
>>> Thanks.
>>>
>>> Chirag
>>>
>>>
>>>
>>>
>>> ________________________________
>>>  From: Claus Ibsen <[hidden email]>
>>> To: "[hidden email]" <[hidden email]>
>>> Sent: Monday, 17 February 2014 12:33 PM
>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>
>>>
>>> Hi
>>>
>>> You should likely reuse the consumer template instead. As this FAQ
>>> applies to it too
>>> http://camel.apache.org/why-does-camel-use-too-many-threads-with-producertemplate.html
>>>
>>> On Mon, Feb 17, 2014 at 8:00 AM, Chirag Dewan <[hidden email]> wrote:
>>>> Hi All,
>>>>
>>>> Any views on this?
>>>>
>>>> Actually my previous mail is not understandable. Let me rephrase the issue.
>>>>
>>>> I am using consumer template to consume single file using fileName property. I have a single instance of consumerTemplate and I call doneUoW and stop after consuming the file.
>>>>
>>>> I am running a test case where 10000 files are consumed by consumerTemplate(one file at a time) . After a while JVM goes OOM and crashes. In heap dump,I can see large number of MemoryIdempotentRepository instances. I am not using idempotent=true. I figured out that inProgressRepository also uses this cache.
>>>>
>>>> Is there something I am missing here?
>>>>
>>>> Thanks in advance.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>  From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Friday, 14 February 2014 3:22 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi All,
>>>>
>>>> I have been looking around why a large number of heap is being consumed by MemoryIdempotentRepository. I saw one JIRA as well regarding inProgressRepository not getting cleared. Can that be the reason here?
>>>>
>>>> Also,I am not using consumerTemplate to consume 1000o files. Does that also use inProgressRepository? And when is it cleared from the memory?
>>>>
>>>> Any help is much appreciated.
>>>>
>>>> Thanks!
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>; Chirag Dewan <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:31 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi,
>>>>
>>>> And I am using readLock=changed. So that can be a reason too?
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Chirag Dewan <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 6:07 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi Claus,
>>>>
>>>> Thanks for the quick reply.
>>>>
>>>> It might well be. But there is one thing I would like to get some insight into.
>>>>
>>>> If idempotent=false(default) is used,will the LRU Cache still store the file referrence? Because that is what I can see in my memory dump.
>>>>
>>>> org.apache.camel.util.LRUCache
>>>>   >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>
>>>> I maybe doing something wrong here. Is there anything else I need to do to make sure that file is deleted after it has been processed and not stored in the cache?
>>>>
>>>> Thanks.
>>>>
>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> ________________________________
>>>>
>>>> From: Claus Ibsen <[hidden email]>
>>>> To: "[hidden email]" <[hidden email]>
>>>> Sent: Wednesday, 12 February 2014 5:42 PM
>>>> Subject: Re: OOM issue due to MemoryIdempotentRepository
>>>>
>>>>
>>>> Hi
>>>>
>>>> Some days ago someone else posted about a OOME issue when using Storm.
>>>> It smells like a Storm + Camel issue somewhere.
>>>>
>>>>
>>>>
>>>> On Wed, Feb 12, 2014 at 11:50 AM, Chirag Dewan <[hidden email]> wrote:
>>>>> Hi All,
>>>>>
>>>>> I am using Camel 2.12.1 with JDK 1.7.0_45. I have a FTP consumer route in my application.
>>>>>
>>>>>
>>>>> Now I intent to delete the file after consuming it. So I have delete=true in my route. After consuming around 6000 files(out of 10000,as my test case) JVM goes Out of memory.
>>>>>
>>>>>
>>>>> In my heap dump I can see the following trace as the major memory consumers:
>>>>>
>>>>> java.util.ArrayList
>>>>>  >java.lang.Object
>>>>>
>>>>>     >org.apache.camel.management.DefaultManagementLifecycleStrategy$PreRegisterService
>>>>>
>>>>>         >org.apache.camel.component.file.FileEndpoint
>>>>>
>>>>>             >org.apache.camel.processor.idempotent.MemoryIdempotentRepository
>>>>>                  >org.apache.camel.util.LRUCache
>>>>>                     >org.apache.camel.com.googlecode.concurrentlinkedhashmap.ConcurrentLinkedHashMap
>>>>>
>>>>>
>>>>> So if I am not using indempotent,should it still store the files in the memory? Is this memory repository used,even if I am deleting the files and not storing in memory? Is there a way I can avoid that?
>>>>>
>>>>> I am testing this scenario under Storm as execution environment and with 2GB max heap space.
>>>>>
>>>>> Thanks in advance
>>>>>
>>>>> Chirag Dewan
>>>>
>>>>
>>>>
>>>> --
>>>> Claus Ibsen
>>>> -----------------
>>>> Red Hat, Inc.
>>>> Email: [hidden email]
>>>> Twitter: davsclaus
>>>> Blog: http://davsclaus.com
>>>> Author of Camel in Action: http://www.manning.com/ibsen
>>>> Make your Camel applications look hawt, try: http://hawt.io

>
>>
>>>
>>>
>>>
>>>
>>> --
>>> Claus Ibsen
>>> -----------------
>>> Red Hat, Inc.
>>> Email: [hidden email]
>>> Twitter: davsclaus
>>> Blog: http://davsclaus.com
>>> Author of Camel in Action: http://www.manning.com/ibsen
>>> Make your Camel applications look hawt, try: http://hawt.io
>>
>>
>>
>> --
>> Claus Ibsen
>> -----------------
>> Red Hat, Inc.
>> Email: [hidden email]
>> Twitter: davsclaus
>> Blog: http://davsclaus.com
>> Author of Camel in Action: http://www.manning.com/ibsen
>> Make your Camel applications look hawt, try: http://hawt.io
>
>
>
> --
> Claus Ibsen
> -----------------
> Red Hat, Inc.
> Email: [hidden email]
> Twitter: davsclaus
> Blog: http://davsclaus.com
> Author of Camel in Action: http://www.manning.com/ibsen
> Make your Camel applications look hawt, try: http://hawt.io



--
Claus Ibsen
-----------------
Red Hat, Inc.
Email: [hidden email]
Twitter: davsclaus
Blog: http://davsclaus.com
Author of Camel in Action: http://www.manning.com/ibsen
Make your Camel applications look hawt, try: http://hawt.io